The essence of data science

I usually treat them by using an extremely well established library where someone else has spent the requisite years crying over every stupid edge case of csv reading. Rolling your own csv reader is a bit like encryption. Until someone hands you a file that rejects all sanity and you start fking with regex. Lol.

If you could choose the column order, you could choose a better format, or at least escape correctly.

🤪 as a delimiter

🥦 for end of line

Use comma for delimiter, and escape any comma in the data by enclosing that entry in quotes.

Data: 225 | 2,500 | 450

CSV: 225,"2,500",450

Alt-008

The delimiter isn't really the issue. Its that there are lots and lots of weird edge cases that break reading csvs. If you use commas, at minimum, you need to escape commas in the data, or quote strings that might contain commas... But now you have to deal with the possibility of a quote character or your escape character in the data.

Then you have the fact that csvs can be written with so many different character encodings, mangling special characters where they occur.

Aaand then you have all the issues that come with lack of metadata - good formats will at least tell you the type of data in each column so you dont have to guess them.

Lets see, its also really annoying to include any binary data in a csv, theres no redundancy or parity checks to catch currupted data, and they arent compressed so you need to tack on compression if you want efficient storage, but that means you always have to read the whole csv file for any task.

Oh, that brings me to the joys of modern columnar formats where you can read selected columns super fast without reading the whole file.

Oh god, I really kept going there. Sorry. Its been a year.