Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

These I believe are what is called fixed width files.

If you use pandas, there is a reader for those `pd.read_fwf()`

I’m sure there are many more languages that support this, it’s just the only one where I had a need.



Exactly that, all banks here in Brazil are exporting transactions with this kind of format (Its called CNAB file over here and they are probably using COBOL systems). You have different kinds of CNAB files, CNAB 200 where you have lines with 200 characters, CNAB 400...

Inside those files you have multiple chunks and every line contains information about a transaction. So if you wanna get the transaction amount you would look for all characters between character n10 and n20 for example. (Those positions are fixed and follow the bank template).

We had success parsing it with python/node, but i think spinning up a sqllite instance, reading straight from the file and doing some calculations would be much faster.


Just thinking out loud and assuming you only have this extension and want to do a quick poc: you could try to delegate the preliminary line parsing to awk (which is very powerful and works in a streaming fashion).

CNAB file -> awk -> select from line_read().

Awk could, for example, transform the fixed-width file in a delimited one (I believe this extension does not strictly require json-formatted inputs). You vould also perform pre-filtering of the input data on awk side (see the "/ /" syntax).

If this extension supports reading from stdin, you could build a traditional unix command pipe, otherwise you would have to glue together the two commands with a named pipe.

Depending on your sensibility, this could feel ugly or very elegant. I am confident it would be very fast, not memory bound, and can be made strict and correct.


SQLite also has support for csv (even without this extension) so if you can process your cnab into csv that may be the easiest most supported way to go.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: