CSV is simple but powerful. Values, separated by commas. It’s easy to understand and use.
\nIt is also, by a wide margin, the one that destroys the most data.
\nThat’s not a paradox. It’s cause and effect. A format simple enough that everyone writes their own parser is a format with as many dialects as it has parsers, and CSV’s defining property is that it carries no information about how to read it.
\nRFC 4180 exists. Yakov Shafranovich published it in October 2005, it registers the text/csv media type, and it gives an ABNF grammar.
It is also Informational, not Standards Track. It was written to describe what people were already doing, twenty-odd years after spreadsheets started emitting it. Section 2 says so outright: there is “no formal specification in existence,” and what follows documents “the format that seems to be followed by most implementations.”
\nBy the time someone wrote it down, every spreadsheet, database, and scripting language had already shipped its own interpretation. The RFC didn’t settle anything. It just added one more dialect, with the distinction of having a number.
\nEight years later, RFC 7111 added URI fragments for pointing at a row, column, or cell inside a text/csv file. Also Informational. CSV still had no standard, but you could now cite a specific cell of one.
The obvious answer is “at the newline,” and the obvious answer is wrong, because a quoted field is allowed to contain one.
\nname,notes\nAlice,"line one\nline two"\nBob,fine\nThat’s a valid three-row CSV. Split it on newlines and you get four:
\nnaive split gives 4 lines:\n 'name,notes'\n 'Alice,"line one'\n 'line two"'\n 'Bob,fine'\n\na real CSV parser gives 3 rows:\n ['name', 'notes']\n ['Alice', 'line one\\nline two']\n ['Bob', 'fine']\nEvery head, wc -l, split("\\n"), and shell pipeline that assumes one record per line is wrong on this file. Not wrong on a malformed file. Wrong on a correct one.
This is the single most common CSV bug, and it’s invisible in testing, because your test fixtures don’t have newlines in them until a user pastes an address into a form.
\nIn most of Europe the decimal separator is a comma. 12,50 is twelve and a half euros. Which means a comma cannot also be a field separator, so those locales use semicolons.
Feed a German CSV to an RFC 4180 parser and everything survives, in the sense that nothing throws:
\ninput: produkt;preis\n Kaffee;12,50\n Tee;9,90\n\nparsed as comma-delimited:\n ['produkt;preis']\n ['Kaffee;12', '50']\n ['Tee;9', '90']\n\nparsed as semicolon-delimited:\n ['produkt', 'preis']\n ['Kaffee', '12,50']\n ['Tee', '9,90']\nThe first reading gives you two columns of nonsense with no error. The prices split down the middle of the decimal point. A pipeline that ingests this will happily compute statistics on the number 12 and the number 50.
\nExcel picks the delimiter based on your operating system’s regional settings, which means the same file opens differently on two machines in the same office.
\n1,2,3\n4,5,6\nHeader or data? Nothing in the file says.
\nRFC 4180’s answer is that you put it in the MIME type: text/csv; header=present. Which is a real answer, and also means the information lives outside the file, in a transport layer that gets stripped the moment someone saves the attachment to disk.
So in practice every tool guesses, usually by checking whether the first row looks less numeric than the rest.
\nEverything above is a parsing problem. This one is worse, because the file parses fine and the damage happens after.
\nCSV has no types. Every value is text. So every spreadsheet and dataframe library applies type inference on import, and type inference is lossy.
\nHere’s a file with four columns of identifiers, all of which are strings that happen to be made of digits:
\ngene,zip,card,accession\nSEPT7,02138,4532012345678901,0004928\nMARCH1,01234,4111111111111111,0000071\nRead it with a type-inferring reader and:
\n gene zip card accession\n SEPT7 2138 4532012345678901 4928\nMARCH1 1234 4111111111111111 71\nThe ZIP code 02138 is now 2138. The accession number 0004928 is now 4928. Nobody was asked. Nothing warned. Save that back to CSV and the original values are gone from disk.
Spreadsheets are worse than this, because they store every number as an IEEE 754 double. Microsoft is blunt about the consequence:
\n\n\nExcel has a maximum precision of 15 significant digits, which means that for any number containing 16 or more digits, such as a credit card number, any numbers past the 15th digit are rounded down to zero.
\n
The example they reach for is a card number:
\ntyped into a cell 1234 5678 9087 6543\nExcel shows 1.23E+15\nMicrosoft calls that “truncating numerical data to 15 digits of precision and converting to a number displayed in scientific notation.” Note that this is the vendor describing its own product, not a bug report.
\nThe card number in the file above is also 16 digits. pandas read it back intact. Excel would not.
\nCredit card numbers are 16 digits. Many national ID numbers are longer. They are not numbers in any meaningful sense, they’re strings of digits, and a format with no type information cannot tell the difference.
\nThe best-documented case of this is genomics, because biologists name genes things like SEPT1 and MARCH1 and spreadsheets read those as dates.
In 2016 Ziemann and colleagues screened 35,175 supplementary Excel files from 18 journals covering 2005 to 2015. Among articles containing Excel gene lists, 19.6% had gene names corrupted this way. One in five.
\nA follow-up in 2021, “Gene name errors: Lessons not learned,” found 30.9% across a broader sample drawn from PubMed Central. Worth being careful comparing those two numbers directly, because the second study used a different sampling frame and also detected an additional error category the first one didn’t look for. The honest summary is that the problem did not go away in the five years after being loudly published.
\nThe resolution is the remarkable part. The field did not fix the spreadsheets. It renamed the genes. The HUGO Gene Nomenclature Committee’s 2020 guidelines state that “all symbols that auto-converted to dates in Microsoft Excel have been changed,” giving SEPT1 becoming SEPTIN1 and MARCH1 becoming MARCHF1 as examples.
Human genes were renamed because a file format cannot say what type a column is.
\nCSV isn’t going away, and mostly shouldn’t. It’s readable, streamable, diffable, and every tool on earth reads it.
\nThe practical defenses are short:
\nThe lesson generalizes past CSV, and it’s the same one from the text file post. A format that carries no description of itself pushes that burden onto every reader, and readers guess. Usually well. Occasionally by silently deleting the leading zero from your ZIP code.
\ntext/csv, January 2014\n\n", "date_published": "2026-08-19T10:00:00-05:00", "url": "https://llbbl.blog/2026/08/19/nobody-agrees-what-a-csv.html", "tags": ["Programming","Data","File-formats","Csv","Excel"] } ] }I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @logan@llbbl.blog.
\n