Nobody Agrees What a CSV Is

Wed, 19 Aug 2026 10:00:00 -0500

CSV is simple but powerful. Values, separated by commas. It’s easy to understand and use.

It is also, by a wide margin, the one that destroys the most data.

That’s not a paradox. It’s cause and effect. A format simple enough that everyone writes their own parser is a format with as many dialects as it has parsers, and CSV’s defining property is that it carries no information about how to read it.

There Is No Standard

RFC 4180 exists. Yakov Shafranovich published it in October 2005, it registers the text/csv media type, and it gives an ABNF grammar.

It is also Informational, not Standards Track. It was written to describe what people were already doing, twenty-odd years after spreadsheets started emitting it. Section 2 says so outright: there is “no formal specification in existence,” and what follows documents “the format that seems to be followed by most implementations.”

By the time someone wrote it down, every spreadsheet, database, and scripting language had already shipped its own interpretation. The RFC didn’t settle anything. It just added one more dialect, with the distinction of having a number.

Eight years later, RFC 7111 added URI fragments for pointing at a row, column, or cell inside a text/csv file. Also Informational. CSV still had no standard, but you could now cite a specific cell of one.

Where Does a Record End?

The obvious answer is “at the newline,” and the obvious answer is wrong, because a quoted field is allowed to contain one.

name,notes
Alice,"line one
line two"
Bob,fine

That’s a valid three-row CSV. Split it on newlines and you get four:

naive split gives 4 lines:
   'name,notes'
   'Alice,"line one'
   'line two"'
   'Bob,fine'

a real CSV parser gives 3 rows:
   ['name', 'notes']
   ['Alice', 'line one\nline two']
   ['Bob', 'fine']

Every head, wc -l, split("\n"), and shell pipeline that assumes one record per line is wrong on this file. Not wrong on a malformed file. Wrong on a correct one.

This is the single most common CSV bug, and it’s invisible in testing, because your test fixtures don’t have newlines in them until a user pastes an address into a form.

What’s the Delimiter?

In most of Europe the decimal separator is a comma. 12,50 is twelve and a half euros. Which means a comma cannot also be a field separator, so those locales use semicolons.

Feed a German CSV to an RFC 4180 parser and everything survives, in the sense that nothing throws:

input:  produkt;preis
        Kaffee;12,50
        Tee;9,90

parsed as comma-delimited:
   ['produkt;preis']
   ['Kaffee;12', '50']
   ['Tee;9', '90']

parsed as semicolon-delimited:
   ['produkt', 'preis']
   ['Kaffee', '12,50']
   ['Tee', '9,90']

The first reading gives you two columns of nonsense with no error. The prices split down the middle of the decimal point. A pipeline that ingests this will happily compute statistics on the number 12 and the number 50.

Excel picks the delimiter based on your operating system’s regional settings, which means the same file opens differently on two machines in the same office.

Is the First Row a Header?

1,2,3
4,5,6

Header or data? Nothing in the file says.

RFC 4180’s answer is that you put it in the MIME type: text/csv; header=present. Which is a real answer, and also means the information lives outside the file, in a transport layer that gets stripped the moment someone saves the attachment to disk.

So in practice every tool guesses, usually by checking whether the first row looks less numeric than the rest.

The Part That Destroys Data

Everything above is a parsing problem. This one is worse, because the file parses fine and the damage happens after.

CSV has no types. Every value is text. So every spreadsheet and dataframe library applies type inference on import, and type inference is lossy.

Here’s a file with four columns of identifiers, all of which are strings that happen to be made of digits:

gene,zip,card,accession
SEPT7,02138,4532012345678901,0004928
MARCH1,01234,4111111111111111,0000071

Read it with a type-inferring reader and:

  gene  zip             card  accession
 SEPT7 2138 4532012345678901       4928
MARCH1 1234 4111111111111111         71

The ZIP code 02138 is now 2138. The accession number 0004928 is now 4928. Nobody was asked. Nothing warned. Save that back to CSV and the original values are gone from disk.

Spreadsheets are worse than this, because they store every number as an IEEE 754 double. Microsoft is blunt about the consequence:

Excel has a maximum precision of 15 significant digits, which means that for any number containing 16 or more digits, such as a credit card number, any numbers past the 15th digit are rounded down to zero.

The example they reach for is a card number:

typed into a cell     1234 5678 9087 6543
Excel shows           1.23E+15

Microsoft calls that “truncating numerical data to 15 digits of precision and converting to a number displayed in scientific notation.” Note that this is the vendor describing its own product, not a bug report.

The card number in the file above is also 16 digits. pandas read it back intact. Excel would not.

Credit card numbers are 16 digits. Many national ID numbers are longer. They are not numbers in any meaningful sense, they’re strings of digits, and a format with no type information cannot tell the difference.

The Gene Name Problem

The best-documented case of this is genomics, because biologists name genes things like SEPT1 and MARCH1 and spreadsheets read those as dates.

In 2016 Ziemann and colleagues screened 35,175 supplementary Excel files from 18 journals covering 2005 to 2015. Among articles containing Excel gene lists, 19.6% had gene names corrupted this way. One in five.

A follow-up in 2021, “Gene name errors: Lessons not learned,” found 30.9% across a broader sample drawn from PubMed Central. Worth being careful comparing those two numbers directly, because the second study used a different sampling frame and also detected an additional error category the first one didn’t look for. The honest summary is that the problem did not go away in the five years after being loudly published.

The resolution is the remarkable part. The field did not fix the spreadsheets. It renamed the genes. The HUGO Gene Nomenclature Committee’s 2020 guidelines state that “all symbols that auto-converted to dates in Microsoft Excel have been changed,” giving SEPT1 becoming SEPTIN1 and MARCH1 becoming MARCHF1 as examples.

Human genes were renamed because a file format cannot say what type a column is.

What To Do About It

CSV isn’t going away, and mostly shouldn’t. It’s readable, streamable, diffable, and every tool on earth reads it.

The practical defenses are short:

Quote everything. It’s never wrong and it removes a whole class of ambiguity.
Treat identifiers as strings explicitly at the point of import. Every serious CSV reader lets you pin column types; use it.
Never round-trip through a spreadsheet if the data contains identifiers. Opening and saving is a lossy operation.
Say what you mean out of band. Delimiter, encoding, header presence, quoting style. The file will not.
Use something else when you can. Parquet and even JSON Lines carry types. If the consumer is a program rather than a person, the readability argument for CSV mostly evaporates.

The lesson generalizes past CSV, and it’s the same one from the text file post. A format that carries no description of itself pushes that burden onto every reader, and readers guess. Usually well. Occasionally by silently deleting the leading zero from your ZIP code.

Sources

RFC 4180 — the Informational spec that documents CSV rather than defining it
RFC 7111 — URI fragment selectors for text/csv, January 2014
Ziemann et al., “Gene name errors are widespread in the scientific literature” — Genome Biology, 2016; the 19.6% figure (free full text)
Abeysooriya et al., “Gene name errors: Lessons not learned” — PLOS Computational Biology, 2021; the 30.9% follow-up
Bruford et al., “Guidelines for human gene nomenclature” — Nature Genetics, 2020; the renaming (free full text)
Microsoft on Excel’s floating-point precision — Excel follows IEEE 754 and stores 15 digits of precision
Microsoft on leading zeros and large numbers — “any numbers past the 15th digit are rounded down to zero,” with a credit card as the example
Microsoft on importing and exporting text files — the CSV list separator comes from Windows Region settings

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @logan@llbbl.blog.

Excel on LLBBL Blog