Text
-
There Is No Such Thing as a Text File
Last time I took apart PNG, which opens it’s file with eight bytes whose entire job is to announce “I am a PNG”.
A text file opens with nothing. No signature, no header, no length field, no version, no metadata. It is bytes, and then it stops.
So this post is the opposite of the last one. Instead of walking a structure, we’re going to look at what happens when there isn’t one.
POSIX Says It Out Loud
Start with the standard. POSIX defines a text file in §3.403:
A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the
<newline>character. Although POSIX.1-2017 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files.Read that bolded part again. The standard defines the term and then tells you the system doesn’t enforce it.
Nothing in the filesystem records “this is text.” There’s no flag on the inode, no attribute, nothing in the directory entry. The
.txtextension is a hint to humans and to Windows. “Text file” is not a property a file has. It’s a claim the reader makes about the bytes, and every tool makes it slightly differently.
The Bytes Don’t Know What They Mean
A file stores bytes. Turning bytes into characters requires an encoding, and the encoding is not in the file.
Here are the same five characters,
Héllo, in several encodings:ascii FAILS: ordinal not in range(128) latin-1 5 bytes 48 e9 6c 6c 6f cp1252 5 bytes 48 e9 6c 6c 6f utf-8 6 bytes 48 c3 a9 6c 6c 6f utf-16 12 bytes ff fe 48 00 e9 00 6c 00 6c 00 6f 00 utf-16-be 10 bytes 00 48 00 e9 00 6c 00 6c 00 6f utf-32 24 bytes ff fe 00 00 48 00 00 00 e9 00 00 00 ...Five characters. Anywhere from 5 to 24 bytes. Nothing in any of those files says which one you’re looking at.
When you open a file in your editor and it looks right, that’s your editor guessing correctly. When you get
caf√©instead ofcafé, that’s your editor guessing wrong. The file never changed.
The Fight Over the Eighth Bit
ASCII was standardized as a 7-bit code: 128 values,
0x00through0x7F. Thirty-three control codes, ninety-five printable characters, and that was the whole world if the world spoke English.Bytes have eight bits though, so there were another 128 values sitting there unused. Everyone grabbed them, and everyone grabbed them differently.
ISO 8859-1 (Latin-1) claimed
0xA0–0xFFfor Western European letters and reserved0x80–0x9Ffor a second set of control codes nobody used. Microsoft looked at those 32 wasted slots and put printable punctuation there instead, creating Windows-1252. That’s where the curly quotes and the em dash live:CP1252 text : It's "fine" — really CP1252 bytes : 49 74 27 73 20 93 66 69 6e 65 94 20 97 20 72 65 61 6c 6c 79 UTF-8 bytes : 49 74 27 73 20 e2 80 9c 66 69 6e 65 e2 80 9d 20 e2 80 94 ...Byte
0x93is a left curly quote in CP1252 and a control character in strict Latin-1. This is why pasting from Word into a system expecting Latin-1 produces garbage: the bytes are legal, they just mean nothing there.Mojibake is exactly this, and it’s completely deterministic:
original text : café as UTF-8 bytes : 63 61 66 c3 a9 read as CP1252 : caféc3 a9is one character in UTF-8 and two characters in CP1252. Both readings are valid. Only one is what you meant.It could have been worse. IBM’s EBCDIC, still running on mainframes, isn’t an ASCII superset at all:
'A' ASCII 0x41 EBCDIC 0xc1 'a' ASCII 0x61 EBCDIC 0x81 ' ' ASCII 0x20 EBCDIC 0x40And the letters aren’t even contiguous.
Iis0xc9,Jis0xd1, with a gap in between. Sorting strings by byte value, which works fine in ASCII, silently produces wrong output in EBCDIC.
Why UTF-8 Won
UTF-8 encodes a character in one to four bytes. ASCII characters keep their single-byte values, so every ASCII file is already a valid UTF-8 file. That backward compatibility gets most of the credit, but the more interesting property is the bit pattern:
'A' U+0041 1 byte 41 01000001 'é' U+00E9 2 bytes c3 a9 11000011 10101001 '€' U+20AC 3 bytes e2 82 ac 11100010 10000010 10101100 '🙂' U+1F642 4 bytes f0 9f 99 82 11110000 10011111 10011001 10000010Look at the leading bits. A single-byte character starts with
0. A multi-byte character starts with110,1110, or11110, where the number of leading 1s is the total byte count. Every continuation byte starts with10, and nothing else does.That makes UTF-8 self-synchronizing. Drop into the middle of a file at a random offset and you can tell immediately whether you’re mid-character, and walk backwards a byte or two to find the boundary. You do not need to have read the file from the beginning.
Compare that to UTF-16, where you must know the byte order and must have tracked whether you’re on an even or odd boundary. UTF-8 made encoding a local property instead of a global one, and that’s why it took over.
The BOM
Multi-byte encodings have a byte order problem: is
00 48the characterU+0048orU+4800? The Byte Order Mark solves it by puttingU+FEFFat the start of the file, so a reader can look at the first two bytes and work out the endianness.utf-16 ff fe 68 69 ... (little-endian) utf-16-le 68 00 69 00 (no BOM, you'd better know) utf-8-sig ef bb bf 68 69 (UTF-8 "BOM") utf-8 68 69 (no BOM)UTF-8 has no byte order to mark, because its unit is one byte. The UTF-8 BOM is not a byte order mark at all; it’s a three-byte flag saying “this is UTF-8,” and the Unicode Consortium neither requires nor recommends it.
It also actively breaks things. The kernel identifies a script by looking for
0x23 0x21, the characters#!, at offset zero:no BOM : 23 21 2f 62 69 6e 2f 73 68 0a -> #!/bin/sh with BOM: ef bb bf 23 21 2f 62 69 6e 2f -> not a scriptSame for JSON parsers, CSV importers, and anything else that expects a specific first byte. If you have ever seen a shell script fail with a cryptic error on a line that looks correct, this is a candidate.
Lines Are a Convention Too
There is no line structure in a text file. There’s a byte that tools agree means “line break,” and even that isn’t agreed on.
The split is a hardware inheritance. A teletype needed two separate mechanical actions to start a new line: carriage return (
0x0D) moved the print head back to the left margin, and line feed (0x0A) advanced the paper by one row. Two actions, two control codes.Then everyone picked differently. Unix chose LF alone. MS-DOS, and Windows after it, kept both as CRLF. Classic Mac OS used CR alone. Those choices are still with us thirty years later, and they’re the reason
.gitattributesexists.And then there’s the trailing newline, which people argue about without realizing the standard already answered it. POSIX §3.206:
A line is a sequence of zero or more non-
<newline>characters plus a terminating<newline>character.The newline is part of the line, not a separator between lines. A file whose last byte isn’t a newline doesn’t have a final line. POSIX §3.195 has a name for what it has instead: an incomplete line.
That definition has teeth:
$ wc -l lf.txt nofinal.txt 2 lf.txt 1 nofinal.txtBoth files contain the text
oneandtwo. The first ends with a newline, the second doesn’t.wc -lcounts newline bytes, so the second file reports one line despite visibly having two.This is also what git’s
\ No newline at end of filemarker means. It isn’t a style complaint. Git is telling you the last line is incomplete by the POSIX definition, which matters because otherwise appending a line would silently modify the existing last line rather than adding a new one.
How Tools Guess
Since nothing declares itself, every tool that needs to know applies a heuristic. The dominant one is: does it contain a NUL byte?
That test exists because C strings are NUL-terminated, so a NUL in the middle of what claims to be text means something is off. It’s a good heuristic. It’s also wrong in two ways worth knowing about.
Git’s version is
buffer_is_binary()inxdiff-interface.c, and it doesn’t scan the whole file. It caps at a constant:#define FIRST_FEW_BYTES 8000So the check is “is there a NUL in the first 8000 bytes.” A file with clean text for 10KB and a NUL after that is text as far as git is concerned. The cutoff is a performance tradeoff, and it means binary-ness is decided by a sample, not a proof.
The second problem is bigger.
plain ascii NUL present: False -> text utf-8 with emoji NUL present: False -> text has a NUL byte NUL present: True -> BINARY utf-16 text NUL present: True -> BINARYUTF-16 encodes ASCII characters as the character byte plus a NUL. Any UTF-16 file that’s mostly English is roughly half NUL bytes. So git does this to a perfectly valid text file:
$ git diff --cached --stat lf.txt | 2 ++ utf16.txt | Bin 0 -> 24 bytesBin. Git will not diff it, will not merge it, and will not show it in review. The heuristic isn’t detecting text, it’s detecting C-string-safety, and those aren’t the same question.file(1)is more thorough, and it shows how much the BOM is doing:$ file utf16.txt utf16_bom.txt lf.txt utf16.txt: data utf16_bom.txt: Unicode text, UTF-16, little-endian text lf.txt: ASCII textIdentical text content in the first two files. The only difference is two leading bytes. Without them
filegives up and calls itdata; with them it identifies the encoding exactly. For a format with no header, a BOM is the closest thing to one that exists.Under the hood
fileis doing real work rather than one heuristic.src/encoding.ccarries a 256-entry table classifying every byte value as never-valid-in-text, ASCII, ISO-8859, or extended ASCII, plus a dedicated UTF-8 state machine that rejects invalid sequences. It then tries candidate encodings in order: ASCII, UTF-7, UTF-8 with BOM, UTF-8, UTF-32, UTF-16, Latin-1, extended ASCII, and finally EBCDIC. That ordering is a nice fossil record of which encodings are still worth guessing first.
Why This Matters
Nearly every format developers work in daily is a convention layered on this substrate. Source code, JSON, YAML, TOML, CSV, Markdown, config files, logs. All of them inherit these problems, and none of them can fully escape them, because the layer underneath has no way to describe itself.
That’s the tradeoff. A format with no header can’t tell you anything about itself, which is exactly why it has outlived every format that could. PNG will be readable as long as someone maintains a PNG decoder. A text file is readable as long as someone remembers what bytes are.
Next in the series: Markdown, which is a text file plus a set of conventions that nobody fully agrees on.
Sources
- POSIX.1-2017 Base Definitions, Chapter 3 — §3.206 Line, §3.195 Incomplete Line, §3.403 Text File
- RFC 3629 — the UTF-8 specification and its byte patterns
- RFC 2046 §4.1 — the
text/plainmedia type - Unicode FAQ on UTF-8, UTF-16, and the BOM — the Consortium’s own guidance on why not to use a UTF-8 BOM
buffer_is_binary()in git’sxdiff-interface.c— the NUL check and theFIRST_FEW_BYTEScutoffsrc/encoding.cin thefileproject — the text-character table and encoding-guessing order behindfile(1)
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].