File-formats
-
Word Documents Used to Be Filesystems
Last post ended on a promise: a
.docxis a ZIP file, and you already know how ZIP works.That’s true, and it’s the smaller half of the story. The interesting part is what
.docxreplaced, because the old.docformat was doing something strange. It wasn’t a document. It was a filesystem with a document living inside it.
A Filesystem in a File
Here’s a real
.docfrom 2015, 10,240 bytes. The first eight bytes:d0 cf 11 e0 a1 b1 1a e1That’s the Compound File Binary Format signature, also called OLE2.
file(1)recognizes it and doesn’t even mention Word:$ file "rich text.doc" Composite Document File V2 Document, Little Endian, Os: Windows, Version 1.0, Code page: -535, Revision Number: 0, Create Time/Date: Thu Dec 10 13:38:22 2015“Composite Document File” is the honest description. CFBF is a container that implements directories and files, called storages and streams, inside a single flat file. It has a File Allocation Table. It has sectors. If that sounds like FAT16, that’s because it’s the same idea, scaled down to live inside one file on a real filesystem.
Cracking this one open gives:
106 bytes CompObj 20 bytes Ole 116 bytes DocumentSummaryInformation 312 bytes SummaryInformation 2411 bytes 1Table 3620 bytes WordDocument sector size : 2^9 = 512 bytes mini sector size : 2^6 = 64 bytesTwo sector sizes, because a 512-byte sector is wasteful for a 20-byte stream. Streams under 4,096 bytes get allocated out of a separate mini-FAT in 64-byte units. There is a fragmentation strategy inside your Word document.
The
WordDocumentstream is the main event, and it opens with a File Information Block whose magic number is0xA5EC:WordDocument stream: 3620 bytes FIB magic (wIdent) = 0xA5ECNone of this is the text yet. This is all container.
The Text Is Not in Order
You’d expect the document’s text to sit in the
WordDocumentstream in reading order. It doesn’t. It sits there in edit order, and a separate structure called a piece table says how to reassemble it.The piece table is a list of descriptors, each saying “characters at logical position X through Y live at physical offset Z.” Reading a
.docmeans walking that table and gathering fragments scattered through the stream.Why build it that way? Because of a feature called Fast Save, and because in 1990 writing to disk was slow. When you edited a document, Word didn’t rewrite the file. It appended your new text to the end of the stream and updated the piece table to point at it. Saving a one-word change to a 200-page document meant writing a few dozen bytes instead of a few hundred kilobytes.
That’s a good optimization. It has an obvious and terrible consequence.
The old text is still in the file. Deleting a paragraph removed it from the piece table, not from the stream. The bytes stayed exactly where they were, unreferenced, invisible in Word, and completely readable in a hex editor.
Microsoft documented this themselves, in a knowledge base article about minimizing metadata in Word documents: “Because of the design of the FastSave feature, text that you delete from a document may remain in the document, even after you save the document.” The recommended fix was to go into Options and clear the “Allow fast saves” check box. From Word 97 SR-1 onward they turned it off by default.
For years, “open the document in a text editor and scroll” was a functioning technique for reading text someone believed they had deleted. Every organization circulating Word files was potentially shipping its own edit history.
The piece table itself has a respectable pedigree. Charles Simonyi brought the technique to Microsoft from Xerox PARC’s Bravo editor, and it’s an elegant way to represent an editable buffer. It’s still how many text editors model documents in memory. The mistake wasn’t the data structure. The mistake was persisting the whole scratch buffer to disk and shipping it to other people.
Then It Became a ZIP of XML
Office 2007 replaced all of it with the Open Packaging Conventions: ECMA-376, later ISO/IEC 29500. A
.docxis a ZIP archive containing XML.Every
.docxopens with the same four bytes:50 4b 03 04 <- PK\x03\x04, a ZIP local file headerPK. Phil Katz’s initials, from the last post, sitting at byte zero of every Word document written since 2007.Unzip one and the structure is legible:
[Content_Types].xml _rels/.rels word/document.xml word/_rels/document.xml.rels word/styles.xml word/settings.xml word/fontTable.xml word/theme/theme1.xml docProps/core.xml docProps/app.xmlword/document.xmlholds the text.[Content_Types].xmlmaps each part to a MIME type._rels/.relsis a relationship graph saying which part is the main document and how the parts connect. The whole thing is a tiny website, zipped.The text itself is WordprocessingML:
<w:p> <w:r> <w:t>Hello, World!</w:t> </w:r> </w:p>A paragraph containing a run containing text. Verbose, but you can read it, and more importantly a program you wrote in an afternoon can read it. That is important when building foundational file formats that outlive the creators.
Extracting text from a
.docmeant implementing a filesystem and a piece table. Extracting text from a.docxmeans unzipping and finding<w:t>elements.The XML contains the document, not the document’s history. Deleted text is deleted.
XML Did Not Mean Simple
It would be tidy to end on “and then it got clean.” The specification runs to several thousand pages, and the ISO fast-track that pushed it through in 2008 was contentious enough to deserve its own post.
What matters here is the shape it settled into. The standard shipped split in two: Strict, the clean format, and Transitional, which carries the legacy baggage forward so documents converted from the binary era still render correctly.
Guess which one nearly everything emits.
Open a Transitional document’s settings and you find a
<w:compat>block. Its children are a museum:w:truncateFontHeightsLikeWP6 WordPerfect 6 w:suppressTopSpacingWP WordPerfect w:lineWrapLikeWord6 Word 6 w:autoSpaceLikeWord95 Word 95 w:footnoteLayoutLikeWW8 Word 97 w:useWord97LineBreakRules Word 97 w:mwSmallCaps Mac WordEvery one of those is a flag asking the renderer to reproduce how a specific piece of 1990s software behaved. Not what the format should do. What Word 6 did do, quirks included. Implementing this correctly means emulating applications whose behavior was never written down anywhere.
The bugs were load-bearing, so they got standardized. The format stopped being a filesystem, but it did not stop being a thirty-year-old application’s memory dumped to disk. It just picked a more legible way to write it down.
Which is, in fairness, an enormous improvement. You can read the file now. You just can’t read all of it quickly.
Sources
- MS-CFB: Compound File Binary Format — Microsoft’s spec for the OLE2 container
- MS-DOC: Word Binary File Format — the FIB, the piece table, and the stream layout
- ECMA-376 — Office Open XML, the basis for
.docx, and free to download. This is the same specification ISO published as ISO/IEC 29500, so read it here rather than paying ISO for the identical text - Library of Congress format description for OOXML — preservation notes and format history
- KB Q223790: WD97: How to Minimize Metadata in Word Documents — the fast-save warning, archived; Microsoft no longer hosts it
w:compatschema reference — the full list of compatibility settings, browsable without downloading the spec
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
ZIP Files Are Read Backwards
Every format in this series so far reads front to back. PNG starts with a signature and you walk chunks in order. A text file is bytes from the beginning. Markdown parsers scan line by line, top to bottom.
ZIP reads backwards. The index is at the end of the file, and a reader is expected to seek to the end first and work its way back.
That one decision explains almost everything strange about ZIP, including a few things that look like bugs and one thing that is definitely a bug.
The Index Lives at the End
Here’s a real ZIP containing two small text files. 241 bytes total. Scanning it for the four-byte record signatures gives the whole layout:
offset bytes ascii record 0 50 4b 03 04 P K . . Local File Header <- hello.txt 53 50 4b 03 04 P K . . Local File Header <- second.txt 108 50 4b 01 02 P K . . Central Directory Header 163 50 4b 01 02 P K . . Central Directory Header 219 50 4b 05 06 P K . . End of Central DirectoryEvery one of those starts with the same two bytes.
0x50is decimal 80, which isPin ASCII.0x4bis decimal 75, which isK.PK. Phil Katz, who wrote PKZIP in 1989, put his initials in the first two bytes of every structure in the format, and they are still there in every.docx,.jar, and.epubon your machine.The two bytes after
PKare the record type:03 04for a local file header,01 02for a central directory entry,05 06for the end-of-central-directory record. Those aren’t printable characters, which is deliberate. A four-byte constant made of two readable letters and two control bytes is unlikely to appear by accident in text, and easy to spot by eye in a hex dump.The last 22 bytes are the End of Central Directory record, and it’s the entry point:
50 4b 05 06 00 00 00 00 02 00 02 00 6f 00 00 00 6c 00 00 00 00 00 signature 0x06054b50 total CD records 2 central dir size 111 bytes central dir offset 108 comment length 0A reader opens the file, jumps to the end, finds that record, reads “the index is at offset 108,” seeks there, and reads the catalog. Listing the contents of a 4 GB archive touches a few hundred bytes.
Note that each file appears twice: once as a Local File Header immediately before its compressed data, and once as an entry in the Central Directory at the end. Hold that thought.
Why Would You Do This?
Because in 1989 you were writing to a floppy disk, and often to a floppy disk that wasn’t big enough.
If the index goes at the front, you have to know everything about every file before you write the first byte: how many files, how big each one compresses to, where each one lands. That means compressing everything to a temporary location, then writing the header, then copying it all back. On a machine with 640K of RAM and two floppy drives, that’s brutal.
Put the index at the end and you can stream. Compress a file, write it, remember where it went. Compress the next one. When you run out of files, write down everything you remembered. One pass, no temporary copy, and you never needed to know the total size in advance.
TAR solved the same problem by having no index at all, which is why
tarhas to read an entire archive to find one file, and why you cannot randomly access a.tar.gz. ZIP got both streaming writes and random-access reads. That’s the trade that made it win.
The Backwards Scan Is Fuzzier Than It Sounds
The EOCD record is 22 bytes, so you’d think a reader could just read the last 22 bytes and be done.
It can’t, because the record ends with a variable-length archive comment of up to 65,535 bytes. The signature isn’t at a fixed offset from the end of the file. So a reader has to seek near the end and scan backwards looking for the four-byte signature, potentially across 65,557 bytes.
Searching for a magic number is not the same as knowing where a structure is. If those four bytes happen to appear inside the comment, or inside compressed data near the end of the file, a naive parser can lock onto the wrong one. Different implementations pick different candidates when there’s more than one. This is a recurring source of “this archive opens in one tool and not another.”
You Can Put Anything in Front of a ZIP
If a reader finds the archive by scanning backwards from the end, then whatever sits at the front of the file is not the reader’s problem.
Take a valid 69-byte PNG, take the 241-byte ZIP, and concatenate them with
cat. No special tooling:$ file polyglot.png polyglot.png: PNG image data, 1 x 1, 8-bit/color RGB, non-interlaced $ unzip -l polyglot.png Length Date Time Name --------- ---------- ----- ---- 12 08-10-2026 16:44 hello.txt 13 08-10-2026 16:44 second.txt --------- ------- 25 2 filesOne 310-byte file. An image viewer reads the PNG signature at byte 0 and renders an image. An archive tool scans backwards, finds the EOCD, and extracts two files. Both are correct. Neither is being fooled by a trick; they’re each doing exactly what their format says to do.
This is the mechanism behind self-extracting archives, where the front of the file is a real executable and the back is a real ZIP. The same property is why “GIFAR” attacks worked: a file that a server accepted as a harmless image was loaded by Java as an archive of classes.
It also means the offsets inside the Central Directory are relative to the start of the archive, not the start of the file, and readers have to work out that difference. Prepending data shifts everything, and well-behaved parsers cope by computing the delta between where the EOCD says the directory should be and where it found it.
Two Indexes, One File
Back to that detail from earlier: every file’s name and metadata are stored twice, in the Local File Header and again in the Central Directory.
Nothing enforces that they agree.
Here’s the same archive with only the Central Directory copy of the first filename patched from
hello.txttoBOGUS.txt. The local header is untouched:$ unzip -l mismatch.zip Length Date Time Name --------- ---------- ----- ---- 12 08-10-2026 16:44 BOGUS.txt 13 08-10-2026 16:44 second.txt local file header at offset 0 still says: hello.txtThe archive is not corrupt.
unziplists it happily. It just contains two different answers to “what is this file called,” and which one you get depends on which structure your parser decided to trust.Now imagine two programs reading the same archive, one checking a signature and the other extracting files. That’s the Android “Master Key” bug from 2013, and the detail is better than the summary.
An APK is a ZIP. The attacker puts two entries in it, both named
classes.dex. Android’s Java verifier loaded entries into a map keyed by filename, so a duplicate name overwrote the earlier one and the last entry was the one whose signature got checked. The native installer used a hash table with linear probing that didn’t replace on collision, so the first entry was the one that got loaded and run. Plant malicious code first, legitimately signed code second, and the device verifies one file and executes the other.A second bug the same year came from the same “two readings, one file” family, via a signed integer. The extra-field length is a 16-bit value, and the Java code read it signed. A length of 65,533 (
0xFFFD) sign-extends to −3. Since the offset of the compressed data is computed by adding that length, a negative value moves the read pointer backward into the header region instead of forward past it.The lesson generalizes past ZIP. Any format that stores the same fact twice has to decide what happens when the copies disagree, and “the spec doesn’t say” is the same answer as “attackers decide.”
Offsets Are Just Numbers
The Central Directory locates each file by offset. Nothing in the format says two entries can’t point at the same bytes.
The classic zip bomb didn’t need that.
42.zipis 42 kilobytes of archives nested five layers deep, sixteen at each layer, unpacking to roughly 4.5 petabytes. The defense is obvious once you’ve seen it: cap recursion depth, don’t auto-extract nested archives.David Fifield’s 2019 construction doesn’t recurse at all. It expands in a single pass, so depth limits are irrelevant. The trick is overlap: many Central Directory entries reference one shared kernel of compressed data, and each entry’s compressed stream uses DEFLATE’s stored-block mode to quote the next entry’s local file header as literal bytes. Entries nest inside each other, and output grows quadratically against input.
He published several, and they aren’t interchangeable:
File Compressed Uncompressed Ratio Needs Zip64 zbsm.zip42 KB 5.5 GB ~130,000:1 No zblg.zip10 MB 281.4 TB ~28,000,000:1 No zbxl.zip46 MB 4.5 PB ~98,000,000:1 Yes The Zip64 requirement on the largest one matters, because not every reader supports Zip64, which makes the merely-enormous version the more portable weapon.
None of these are malformed files. Every one is a valid archive that a conforming parser is supposed to accept. The format allows two entries to describe the same bytes, and no rule anywhere says the total uncompressed size has to bear any relationship to the file you’re holding.
Everything Is Secretly a ZIP
Once you know the structure, you start recognizing it:
.docx,.xlsx,.pptxare ZIP archives of XML.jar,.war,.apkare ZIP archives of class files and resources.epubis a ZIP of XHTML.odt,.odsare ZIP of XML again
That’s not a coincidence or a hack. ISO/IEC 21320-1, “Document Container File,” defines a constrained ZIP profile for exactly this use. It narrows the format so a
.docxreader doesn’t have to implement all of ZIP’s accumulated history: compression must be stored or deflated and nothing else, and the various encryption and digital-signature mechanisms in the original spec are all forbidden.It’s a narrowing, not a rewrite. Zip64 version 1 is still permitted, for instance; only version 2 is ruled out. The profile is best understood as a list of the parts of ZIP that turned out to be a bad idea.
Which means the next post in this series is mostly about a ZIP file with XML inside it. You already know half of how a Word document works.
Sources
- PKWARE APPNOTE.TXT — the original and still-authoritative ZIP specification, currently version 6.3.10; §4.3.16 defines the end of central directory record
- ISO/IEC 21320-1:2015 — the constrained ZIP profile used by document formats. Fair warning, this one is a paid ISO standard; the catalog page tells you what it covers but you cannot read the text without buying it
- Library of Congress format description for ZIP — history and preservation notes
- David Fifield: A Better Zip Bomb — the overlapping-stream construction
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Markdown Is Not a Format, It's an Argument
I’ve covered PNG and text files, and now it’s time for Markdown, which can be thought of as a philosophy of formatting or a lifestyle of text documents more so than an actually well defined file format. It has structure, and it has specifications, plural, and nothing agrees.
Here is three lines of Markdown run through five parsers:
INPUT: "- outer\n - inner\n" Python-Markdown <ul> <li>outer</li> <li>inner</li> </ul> markdown2 <ul> <li>outer <ul> <li>inner</li> </ul></li> </ul> mistune <ul> <li>outer<ul> <li>inner</li> </ul> </li> </ul> marko (CommonMark) <ul> <li> outer<ul> <li>inner</li> </ul> </li> </ul> cmark-gfm (GitHub) <ul> <li>outer <ul> <li>inner</li> </ul> </li> </ul>Five parsers, five different results. Most of that is cosmetic whitespace, but look at the first one: Python-Markdown produced a flat list. The nesting is gone. That’s not a formatting difference, that’s a different document.
The Original Spec Was an Essay
John Gruber released Markdown in March 2004, along with a Perl script called
Markdown.pl. The design goal was stated plainly:The overriding design goal for Markdown’s formatting syntax is to make it as readable as possible. The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions.
That goal was met, and it’s why we’re all still using it twenty years later. The syntax borrowed from conventions people had already invented for plain text email and Usenet:
=and-underlines from Setext,#headers from atx,>quoting from Usenet,*for emphasis from Textile and reStructuredText. None of it was new. That was the point.What Markdown shipped without was a grammar. The specification was English prose describing the syntax with examples, and the tiebreaker for anything the prose didn’t cover was “whatever
Markdown.pldoes.” A Perl script full of regular expressions became the definition of the format by default.That works fine until someone writes a second implementation.
Where the Prose Ran Out
The ambiguities weren’t exotic. They were things you hit in the first week:
How much indentation nests a list? Two spaces? Four? One tab? The original prose didn’t say clearly, and the answer interacts with the rule that four spaces means a code block.
What happens inside raw HTML? If you write a
<div>and put Markdown inside it, does the Markdown get processed? Gruber’s implementation had behavior; the prose didn’t specify it.When does a
*open emphasis versus just being an asterisk? Ina * b * c, are those multiplication signs or emphasis delimiters?Do underscores work inside words? This one bites daily:
INPUT: "snake_case_variable" Python-Markdown <p>snake_case_variable</p> markdown2 <p>snake<em>case</em>variable</p> mistune <p>snake_case_variable</p> marko (CommonMark) <p>snake_case_variable</p> cmark-gfm (GitHub) <p>snake_case_variable</p>markdown2 italicizes your variable name. Every other parser leaves it alone. Both are defensible readings of a spec that never addressed it.
Or the heading with no space after the hash:
INPUT: "#Heading" Python-Markdown <h1>Heading</h1> markdown2 <h1>Heading</h1> mistune <p>#Heading</p> marko (CommonMark) <p>#Heading</p> cmark-gfm (GitHub) <p>#Heading</p>Half of them give you a heading, half give you a paragraph starting with a hash. This one matters because
#hashtagat the start of a line is a real thing people write.
Everyone Wrote Their Own
With no formal spec, every implementation became a dialect, and the popular ones added features:
- PHP Markdown Extra (Michel Fortin, 2005) added pipe tables, definition lists, footnotes, fenced code blocks, and attribute blocks.
- MultiMarkdown (Fletcher Penney, 2005) added metadata frontmatter, cross-references, citations, and LaTeX export.
- Pandoc Markdown (John MacFarlane, 2006) built a real AST-based parser and added YAML frontmatter, TeX math, grid tables, and citations.
- kramdown (Thomas Leitner, 2009) added inline attribute lists and its own math support.
Each is a superset of a slightly different reading of the original. A document written for one is not guaranteed to render correctly in another, and the failure mode is silent: you don’t get a parse error, you get the wrong document.
CommonMark: Specify the Ambiguity Away
On 3 September 2014, Jeff Atwood announced a spec effort on Coding Horror under the name Standard Markdown, with John MacFarlane as primary author and people from GitHub, Reddit, Stack Exchange, and Meteor involved. The goal was not a new dialect and not a replacement for Gruber’s syntax, but an unambiguous description of what the existing syntax should mean in every case.
The name lasted about a day. That night, by Atwood’s account, Gruber emailed him and MacFarlane privately, called the name “infuriating,” and asked that the project be renamed and the domain taken down. On 4 September, Atwood published a follow-up retitling it Common Markdown, which shortly became the one-word CommonMark.
Worth being precise here, because this story gets retold badly: this was not a trademark action. Gruber holds no registered trademark on “Markdown” and did not invoke one. It was an objection to the name, made in private email, and the only public record of his side is Atwood’s paraphrase. There is no Daring Fireball post about it.
The naming fight is a footnote. The approach is the interesting part. Rather than describing the syntax in prose and hoping, CommonMark defines a parsing algorithm and ships an executable test suite pairing exact input with exact expected HTML, more than 500 examples embedded in the spec document itself. Conformance is not a matter of opinion. You run the tests.
The algorithm works in two passes.
Phase one walks the document line by line and builds block structure. Container blocks (blockquotes, lists, list items) and leaf blocks (headings, code blocks, paragraphs, HTML blocks) get assembled into a tree. Link reference definitions get collected. No inline formatting is considered at all in this phase, which is why block structure always wins: a
>at the start of a line is a blockquote marker regardless of what emphasis you thought you were in the middle of.Phase two walks the text inside leaf blocks and resolves inline structure. This is where emphasis, links, images, code spans, and inline HTML get parsed, using a delimiter stack.
That two-phase split is the single most useful thing to know about Markdown parsing, because it explains most surprising behavior. If your emphasis “leaked” across a list item boundary, it didn’t; blocks were decided before emphasis was ever considered.
The Emphasis Rules Are Hard
Emphasis is the hardest part of the spec, and CommonMark’s solution is a set of flanking rules. A run of
*or_is classified as left-flanking (can open emphasis) or right-flanking (can close it) based on the characters on either side, roughly: a delimiter can open if it’s not followed by whitespace, and can close if it’s not preceded by whitespace, with extra conditions around punctuation.Then there’s a special case for underscores: an
_can open emphasis only if it’s left-flanking and not right-flanking. That single asymmetry is what makessnake_case_variablesafe, because the middle underscores are both left- and right-flanking and are therefore disqualified from opening anything. Asterisks don’t get that rule, which is whysnake*case*variablestill italicizes.This is what “specifying the ambiguity away” costs. The rule isn’t elegant. It exists because real documents contain identifiers, and a spec that italicizes your variable names is wrong no matter how clean its grammar is.
You can see the payoff in the nesting case:
INPUT: "*foo**bar**baz*" Python-Markdown <p><em>foo</em><em>bar</em><em>baz</em></p> everyone else <p><em>foo<strong>bar</strong>baz</em></p>Four parsers agree, and the one that predates the delimiter-stack approach gets it wrong in a way that changes the meaning.
GFM Is a Layer, Not a Fork
GitHub Flavored Markdown is CommonMark plus five extensions, and it’s specified against CommonMark rather than diverging from it:
- Tables, pipe-delimited with alignment colons
- Task lists,
- [ ]and- [x], rendered as checkboxes - Strikethrough,
~~text~~ - Autolinks, bare URLs linkified without brackets
- A raw HTML filter that neutralizes dangerous tags by escaping their opening bracket
That last one is a security control rather than a formatting feature, which tells you something about what it’s like to run a Markdown renderer on user-submitted content at GitHub’s scale.
The extension boundary is visible if you feed the same table to both:
INPUT: | a | b | |---|---| | 1 | 2 | CommonMark <p>| a | b | |---|---| | 1 | 2 |</p> cmark-gfm <table><thead><tr><th>a</th><th>b</th></tr></thead>...Tables are not Markdown. Tables are a GFM extension. CommonMark renders that input as a paragraph containing literal pipe characters, and it is correct to do so.
Tables, footnotes, task lists, strikethrough, frontmatter, math, and Mermaid diagrams are all extensions. None of them are guaranteed anywhere.
What To Do About It
The practical takeaways are short.
Know which parser you’re targeting. “It renders on GitHub” tells you about cmark-gfm, and nothing about your static site generator, your docs pipeline, or someone’s RSS reader.
Prefer the constructs everyone agrees on. Headings with a space after the hash, fenced code blocks, asterisks for emphasis, blank lines between blocks, four-space or consistent nesting. Boring Markdown survives transport.
Don’t rely on parser-specific behavior you discovered by accident. If nesting a list at two spaces works in your tool, that’s your tool, not the format.
There is even a formal way to say which dialect you mean. RFC 7763 registers
text/markdownas a media type, and RFC 7764 defines avariantparameter for exactly this problem:text/markdown; variant=CommonMark text/markdown; variant=GFM text/markdown; variant=OriginalThe standards process looked at Markdown, concluded that saying “this is Markdown” is not specific enough to be useful, and standardized a way to say which Markdown you meant.
That’s the tradeoff Markdown made. PNG picked one answer and enforced it with a checksum. A text file refuses to answer anything. Markdown let a million answers bloom, got adopted everywhere precisely because it was easy to implement badly, and has spent the last decade trying to agree with itself.
I’ll take that trade. But it’s worth knowing that when you write Markdown, you are not writing in a format. You’re writing in a dialect, and hoping the reader speaks it.
Sources
- Daring Fireball: Markdown — Gruber’s original 2004 syntax document and design goals
- CommonMark Specification — the parsing algorithm, emphasis flanking rules, and executable test suite
- CommonMark parsing strategy appendix — the two-phase block/inline design
- GitHub Flavored Markdown Spec — the five extensions, specified against CommonMark
- RFC 7763 and RFC 7764 — the
text/markdownmedia type and the registered dialect variants, both by S. Leonard, March 2016 - Coding Horror: Standard Flavored Markdown and Standard Markdown is now Common Markdown — Atwood’s announcement and the rename a day later
- Daring Fireball: Introducing Markdown — the original 15 March 2004 announcement
tagfilter.cin cmark-gfm — the nine tags GFM’s raw HTML filter neutralizes
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
There Is No Such Thing as a Text File
Last time I took apart PNG, which opens with eight bytes whose entire job is to announce “I am a PNG”.
A text file opens with nothing. No signature, no header, no length field, no version, no metadata. It is bytes, and then it stops.
So this post is the opposite of the last one. Instead of walking a structure, we’re going to look at what happens when there isn’t one.
POSIX Defines the Standard
Start with the standard, in §3.403:
A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the
<newline>character. Although POSIX.1-2017 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files.The standard defines the term and then tells you the system doesn’t enforce it.
Nothing in the filesystem records “this is text.” There’s no flag on the inode, no attribute, nothing in the directory entry. The
.txtextension is a hint to humans and to Windows. “Text file” is not a property a file has. It’s a claim the reader makes about the bytes, and every tool makes it slightly differently.
The Bytes Don’t Know What They Mean
A file stores bytes. Turning bytes into characters requires an encoding, and the encoding is not in the file.
Here are the same five characters,
Héllo, in several encodings:ascii FAILS: ordinal not in range(128) latin-1 5 bytes 48 e9 6c 6c 6f cp1252 5 bytes 48 e9 6c 6c 6f utf-8 6 bytes 48 c3 a9 6c 6c 6f utf-16 12 bytes ff fe 48 00 e9 00 6c 00 6c 00 6f 00 utf-16-be 10 bytes 00 48 00 e9 00 6c 00 6c 00 6f utf-32 24 bytes ff fe 00 00 48 00 00 00 e9 00 00 00 ...Five characters. Anywhere from 5 to 24 bytes. Nothing in any of those files says which one you’re looking at.
When you open a file in your editor and it looks right, that’s your editor guessing correctly. When you get
caf√©instead ofcafé, that’s your editor guessing wrong. The file never changed.
The Fight Over the Eighth Bit
ASCII was standardized as a 7-bit code: 128 values,
0x00through0x7F. Thirty-three control codes, ninety-five printable characters, and that was the whole world if the world spoke English.Bytes have eight bits though, so there were another 128 values sitting there unused. Everyone grabbed them, and everyone grabbed them differently.
ISO 8859-1 (Latin-1) claimed
0xA0–0xFFfor Western European letters and reserved0x80–0x9Ffor a second set of control codes nobody used. Microsoft looked at those 32 wasted slots and put printable punctuation there instead, creating Windows-1252. That’s where the curly quotes and the em dash live:CP1252 text : It's "fine" — really CP1252 bytes : 49 74 27 73 20 93 66 69 6e 65 94 20 97 20 72 65 61 6c 6c 79 UTF-8 bytes : 49 74 27 73 20 e2 80 9c 66 69 6e 65 e2 80 9d 20 e2 80 94 ...Byte
0x93is a left curly quote in CP1252 and a control character in strict Latin-1. This is why pasting from Word into a system expecting Latin-1 produces garbage: the bytes are legal, they just mean nothing there.Mojibake, Japanese for “character transformation,” is exactly this, and it’s completely deterministic:
original text : café as UTF-8 bytes : 63 61 66 c3 a9 read as CP1252 : caféc3 a9is one character in UTF-8 and two characters in CP1252. Both readings are valid. Only one is what you meant.It could have been worse. IBM’s EBCDIC, still running on mainframes, isn’t an ASCII superset at all:
'A' ASCII 0x41 EBCDIC 0xc1 'a' ASCII 0x61 EBCDIC 0x81 ' ' ASCII 0x20 EBCDIC 0x40And the letters aren’t even contiguous.
Iis0xc9,Jis0xd1, with a gap in between. Sorting strings by byte value, which works fine in ASCII, silently produces wrong output in EBCDIC.
Why UTF-8 Won
UTF-8 encodes a character in one to four bytes. ASCII characters keep their single-byte values, so every ASCII file is already a valid UTF-8 file. That backward compatibility gets most of the credit, but the more interesting property is the bit pattern:
'A' U+0041 1 byte 41 01000001 'é' U+00E9 2 bytes c3 a9 11000011 10101001 '€' U+20AC 3 bytes e2 82 ac 11100010 10000010 10101100 '🙂' U+1F642 4 bytes f0 9f 99 82 11110000 10011111 10011001 10000010Look at the leading bits. A single-byte character starts with
0. A multi-byte character starts with110,1110, or11110, where the number of leading 1s is the total byte count. Every continuation byte starts with10, and nothing else does.That makes UTF-8 self-synchronizing. Drop into the middle of a file at a random offset and you can tell immediately whether you’re mid-character, and walk backwards a byte or two to find the boundary. You do not need to have read the file from the beginning.
Compare that to UTF-16, where you must know the byte order and must have tracked whether you’re on an even or odd boundary. UTF-8 made encoding a local property instead of a global one, and that’s why it took over.
The BOM
Multi-byte encodings have a byte order problem: is
00 48the characterU+0048orU+4800? The Byte Order Mark solves it by puttingU+FEFFat the start of the file, so a reader can look at the first two bytes and work out the endianness.utf-16 ff fe 68 69 ... (little-endian) utf-16-le 68 00 69 00 (no BOM, you'd better know) utf-8-sig ef bb bf 68 69 (UTF-8 "BOM") utf-8 68 69 (no BOM)UTF-8 has no byte order to mark, because its unit is one byte. The UTF-8 BOM is not a byte order mark at all; it’s a three-byte flag saying “this is UTF-8,” and the Unicode Consortium neither requires nor recommends it.
It also actively breaks things. The kernel identifies a script by looking for
0x23 0x21, the characters#!, at offset zero:no BOM : 23 21 2f 62 69 6e 2f 73 68 0a -> #!/bin/sh with BOM: ef bb bf 23 21 2f 62 69 6e 2f -> not a scriptSame for JSON parsers, CSV importers, and anything else that expects a specific first byte. If you have ever seen a shell script fail with a cryptic error on a line that looks correct, this is a candidate.
Lines Are a Convention Too
There is no line structure in a text file. There’s a byte that tools agree means “line break,” and even that isn’t agreed on.
The split is a hardware inheritance. A teletype needed two separate mechanical actions to start a new line: carriage return (
0x0D) moved the print head back to the left margin, and line feed (0x0A) advanced the paper by one row. Two actions, two control codes.Then everyone picked differently. Unix chose LF alone. MS-DOS, and Windows after it, kept both as CRLF. Classic Mac OS used CR alone. Those choices are still with us thirty years later, and they’re the reason
.gitattributesexists.And then there’s the trailing newline, which people argue about without realizing the standard already answered it. POSIX §3.206:
A line is a sequence of zero or more non-
<newline>characters plus a terminating<newline>character.The newline is part of the line, not a separator between lines. A file whose last byte isn’t a newline doesn’t have a final line. POSIX §3.195 has a name for what it has instead: an incomplete line.
That definition has teeth:
$ wc -l lf.txt nofinal.txt 2 lf.txt 1 nofinal.txtBoth files contain the text
oneandtwo. The first ends with a newline, the second doesn’t.wc -lcounts newline bytes, so the second file reports one line despite visibly having two.This is also what git’s
\ No newline at end of filemarker means. It isn’t a style complaint. Git is telling you the last line is incomplete by the POSIX definition, which matters because otherwise appending a line would silently modify the existing last line rather than adding a new one.
How Tools Guess
Since nothing declares itself, every tool that needs to know applies a heuristic. The dominant one is: does it contain a NUL byte?
That test exists because C strings are NUL-terminated, so a NUL in the middle of what claims to be text means something is off. It’s a good heuristic. It’s also wrong in two ways worth knowing about.
Git’s version is
buffer_is_binary()inxdiff-interface.c, and it doesn’t scan the whole file. It caps at a constant:#define FIRST_FEW_BYTES 8000So the check is “is there a NUL in the first 8000 bytes.” A file with clean text for 10KB and a NUL after that is text as far as git is concerned. The cutoff is a performance tradeoff, and it means binary-ness is decided by a sample, not a proof.
The second problem is bigger.
plain ascii NUL present: False -> text utf-8 with emoji NUL present: False -> text has a NUL byte NUL present: True -> BINARY utf-16 text NUL present: True -> BINARYUTF-16 encodes ASCII characters as the character byte plus a NUL. Any UTF-16 file that’s mostly English is roughly half NUL bytes. So git does this to a perfectly valid text file:
$ git diff --cached --stat lf.txt | 2 ++ utf16.txt | Bin 0 -> 24 bytesBin. Git will not diff it, will not merge it, and will not show it in review. The heuristic isn’t detecting text, it’s detecting C-string-safety, and those aren’t the same question.file(1)is more thorough, and it shows how much the BOM is doing:$ file utf16.txt utf16_bom.txt lf.txt utf16.txt: data utf16_bom.txt: Unicode text, UTF-16, little-endian text lf.txt: ASCII textIdentical text content in the first two files. The only difference is two leading bytes. Without them
filegives up and calls itdata; with them it identifies the encoding exactly. For a format with no header, a BOM is the closest thing to one that exists.Under the hood
fileis doing real work rather than one heuristic.src/encoding.ccarries a 256-entry table classifying every byte value as never-valid-in-text, ASCII, ISO-8859, or extended ASCII, plus a dedicated UTF-8 state machine that rejects invalid sequences. It then tries candidate encodings in order: ASCII, UTF-7, UTF-8 with BOM, UTF-8, UTF-32, UTF-16, Latin-1, extended ASCII, and finally EBCDIC. That ordering is a nice fossil record of which encodings are still worth guessing first.
Why This Matters
Nearly every format developers work in daily is a convention layered on this substrate. Source code, JSON, YAML, TOML, CSV, Markdown, config files, logs. All of them inherit these problems, and none of them can fully escape them, because the layer underneath has no way to describe itself.
That’s the tradeoff. A format with no header can’t tell you anything about itself, which is exactly why it has outlived every format that could. PNG will be readable as long as someone maintains a PNG decoder. A text file is readable as long as someone remembers what bytes are.
Next in the series: Markdown, which is a text file plus a set of conventions that nobody fully agrees on.
Sources
- POSIX.1-2017 Base Definitions, Chapter 3 — §3.206 Line, §3.195 Incomplete Line, §3.403 Text File
- RFC 3629 — the UTF-8 specification and its byte patterns
- RFC 2046 §4.1 — the
text/plainmedia type - Unicode FAQ on UTF-8, UTF-16, and the BOM — the Consortium’s own guidance on why not to use a UTF-8 BOM
buffer_is_binary()in git’sxdiff-interface.c— the NUL check and theFIRST_FEW_BYTEScutoffsrc/encoding.cin thefileproject — the text-character table and encoding-guessing order behindfile(1)
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
How PNG Actually Stores Your Pixels
I’m starting a series on file formats. Not “here are the ten image formats you should know,” but the actual bytes: what’s in the file, in what order, and why someone decided it should be that way.
Starting with PNG, because it’s the format most developers touch every day and almost nobody has looked inside.
I am likely to cover a few things that other explainer documents have covered, such as chunk structure and chunk types. However, I’d like to dig into some details that are not often mentioned, such as where your pixels went.
A Format Born From a Patent Fight
PNG exists because of a licensing ambush. On 28 December 1994, right in the middle of the holidays, Unisys announced an agreement to start collecting royalties from authors of GIF-supporting software, on the strength of its patent on the LZW compression algorithm that GIF used.
The response was fast. A draft for a replacement format was posted to
comp.graphicson 4 January 1995, one week later. It was originally called PBF, for Portable Bitmap Format, and got renamed to PNG two days after that. The format shipped as a W3C Recommendation in October 1996.Two things about that origin still show in the bytes. The format is aggressively defensive, because it was designed by people who expected files to be mangled in transit. And it is aggressively extensible, because they had just watched a format become unusable for reasons that had nothing to do with its technical design.
Eight Bytes of Paranoia
Every PNG starts with the same eight bytes:
Hexadecimal: 89 50 4E 47 0D 0A 1A 0A ASCII/Ctrl: \x89 P N G \r \n \x1A \nP N Gin the middle is obvious. The other five bytes are a booby trap for 1995-era file transfer, and each one catches a specific failure:0x89has the high bit set. Some 7-bit transfer paths stripped bit 7 from every byte. If that happened, this byte arrives as0x09and the file is detectably wrong on byte one.0x0D 0x0Ais a DOS line ending. A text-mode FTP transfer that “helpfully” converts CRLF to LF mangles it.0x1Ais Ctrl-Z, the MS-DOS end-of-file marker. If youTYPEa PNG at a DOS prompt, output stops here instead of spraying binary at your terminal and leaving it in a weird state.0x0Ais a bare LF, catching the opposite conversion: LF silently expanded to CRLF.
Who would have thought that so many bits were used just to account for line endings in different operating systems? I suppose it’s good to plan ahead when designing a file format.
Everything Is a Chunk
After the signature, a PNG is a flat sequence of chunks. No central directory, no offset table. You read them in order.
Every chunk has the same four-field shape:
Field Size Notes Length 4 bytes Big-endian, counts only the data field Chunk Type 4 bytes Four ASCII letters Chunk Data Length bytes Can be zero-length CRC-32 4 bytes Computed over type and data, not over length Two details worth keeping. The length field is 32 bits but the spec caps values at 2³¹−1, so the high bit is always clear. And the CRC covers the type plus the data but skips the length, which means a corrupted length field is not detected by the chunk’s own checksum.
The chunk type is where PNG does something clever. Those four letters are ASCII, and bit 5 of an ASCII letter is what distinguishes uppercase from lowercase (
Ais0x41,ais0x61). PNG uses that bit in each of the four positions as a flag:Position Uppercase means Lowercase means 1st Critical: decoder must understand it Ancillary: safe to ignore 2nd Public, registered in the spec Private, vendor-specific 3rd Reserved, must be uppercase today (reserved for future use) 4th Unsafe to copy if pixels changed Safe to copy blindly So a decoder that has never heard of
tEXtcan tell from the lowercasetthat skipping it is fine. A decoder hittingIDATsees the uppercaseIand knows it cannot skip it. The capability negotiation is encoded in the name itself, which means you can add chunk types decades later without breaking old readers. This is why APNG could bolt animation onto PNG without a version bump.Four chunk types are critical:
IHDR(header, always first),PLTE(palette),IDAT(the pixels), andIEND(a zero-length terminator).
IHDR Is the Decode Key
IHDRis exactly 13 bytes and it comes first because nothing else can be interpreted without it:- Width (4 bytes) and Height (4 bytes), big-endian
- Bit depth (1 byte): bits per sample, one of 1, 2, 4, 8, 16
- Color type (1 byte): what a pixel is made of
- Compression method (1 byte): always 0
- Filter method (1 byte): always 0
- Interlace method (1 byte): 0 for none, 1 for Adam7
Bit depth and color type together determine everything about the pixel layout, and only certain combinations are legal:
Color type Name Samples per pixel Legal bit depths 0 Greyscale 1 1, 2, 4, 8, 16 2 Truecolor (RGB) 3 8, 16 3 Indexed 1 (a palette index) 1, 2, 4, 8 4 Greyscale + alpha 2 8, 16 6 Truecolor + alpha (RGBA) 4 8, 16 Note the gaps. You cannot have 16-bit indexed color, because a palette holds at most 256 entries and 8 bits already addresses all of them. You cannot have 1-bit RGB, because a “1-bit red sample” isn’t a useful thing. The table isn’t arbitrary; each missing cell is a combination that would be incoherent.
Also note that bit depth is per sample, not per pixel. A bit depth of 16 with color type 6 means 16 bits each for R, G, B, and A: 64 bits per pixel. That’s the “64-bit RGBA” you see in PNG marketing.
Where the Pixels Actually Live
Uncompress all the
IDATdata and concatenate it, and you get a byte stream. That stream is not a grid. It’s a sequence of scanlines, one per image row, top to bottom. And each scanline is:[1 filter type byte][packed sample data for the whole row]That leading byte is not pixel data. It’s a number from 0 to 4 saying which filter was applied to this row.
The sample data is packed with no padding between pixels and no separators. Samples appear in a fixed order within each pixel:
- Greyscale:
grey - Truecolor:
red, green, blue - Indexed:
palette index - Greyscale + alpha:
grey, alpha - Truecolor + alpha:
red, green, blue, alpha
Here is an example PNG, filter type 0 (no filtering) on both rows:
scanline 0: 00 ff 00 00 00 ff 00 00 00 ff ff ff 00 ^^ filter byte ^^^^^^^^ red pixel (ff,00,00) ^^^^^^^^ green pixel (00,ff,00) scanline 1: 00 00 00 00 80 80 80 ff ff ff ff 00 ff ^^ filter byte ^^^^^^^^ black ^^^^^^^^ whiteTwelve bytes of pixel data per row (4 pixels × 3 samples), each prefixed by one filter byte, for 26 bytes of raw stream. The complete file, signature and all four chunks included, is 83 bytes.
The whole model at 8-bit depth: walk the row, emit samples in order, move on. No alignment, no padding, no per-pixel headers.
Below 8 Bits, Pixels Share Bytes
Bit depths of 1, 2, and 4 only apply to greyscale and indexed images. Multiple pixels get packed into a single byte.
These samples are packed into bytes with the leftmost sample in the high-order bits of a byte followed by the other samples for the scanline.
Leftmost pixel goes in the high bits. So for a 12-pixel-wide 1-bit greyscale image:
pixels : 1 1 0 1 0 0 0 1 1 0 1 1 packed bytes : 0xd1 0xb0 11010001 10110000 ^^^^ unusedTwelve pixels need 12 bits, which rounds up to 2 bytes, leaving 4 bits spare at the end. The spec’s language on those leftover bits:
When there are multiple pixels per byte, some low-order bits of the last byte of a scanline may go unused. The contents of these unused bits are not specified.
Scanlines always start on a byte boundary. Row 2 never continues in the leftover bits of row 1’s last byte.
At bit depth 16, each sample is two bytes, most significant byte first. The spec calls it network byte order. On x86 and ARM, which are little-endian, that means every 16-bit sample needs a byte swap on read and on write.
The Filter Byte Is the Whole Trick
Now back to that leading byte on every scanline.
PNG uses DEFLATE, the same algorithm as gzip and zip. If you just DEFLATE’d raw pixels, PNG would compress about as well as gzipping a bitmap, which is to say barely at all. Photographs and gradients don’t repeat exact byte sequences, and LZ77 needs exact repeats.
So before compressing, PNG transforms each scanline into differences from its neighbors. Five filters are available, chosen per scanline:
Type Name Transform 0 None store the byte as-is 1 Sub subtract the byte from the pixel to the left 2 Up subtract the byte from the pixel above 3 Average subtract the average of left and above 4 Paeth subtract whichever of left/above/upper-left is the best predictor All arithmetic is mod 256, which is what makes it reversible without storing a sign. And “the pixel to the left” means the byte at the same position in the previous pixel, so for RGB the red sample is compared against the previous red sample, not against the previous blue.
Take a 16-pixel greyscale gradient stepping by 10:
raw scanline : 00 0a 14 1e 28 32 3c 46 50 5a 64 6e 78 82 8c 96 after Sub filter : 00 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0aSixteen distinct byte values become two. The image is unchanged and the transform is exactly reversible, but LZ77 now sees a run it can encode in almost nothing. On this toy row, DEFLATE produces 28 bytes for the raw version and 15 for the filtered one.
Sixteen bytes is far too small for DEFLATE to stretch its legs, so don’t read that ratio as typical. The point is the entropy collapse: filtering doesn’t compress anything, it rearranges the data so the compressor has something to find.
That per-scanline choice is also why two encoders produce different-sized files from identical pixels. libpng, ImageMagick,
oxipng, andzopflipngall ship different filter-selection heuristics. Same spec, same decoded output, different bytes on disk. Most PNG optimizers are search algorithms over filter choices, not better compressors.
The Compression Pipeline, End to End
Putting it together:
raw pixels -> pack into scanlines (samples in order, sub-byte packing if needed) -> prepend a filter byte per scanline, apply the filter -> DEFLATE the whole concatenated stream (LZ77 + Huffman) -> wrap in a zlib container (RFC 1950) -> split across one or more IDAT chunksA few consequences fall out of that ordering:
The zlib stream spans chunks.
IDATboundaries are arbitrary. A decoder must concatenate everyIDATpayload and then decompress; decompressing them individually fails. Encoders split them for streaming, not for structure.The Adler-32 checksum in the zlib wrapper covers filtered bytes, not your original pixels. It validates decompression, not image fidelity. The per-chunk CRC-32 is what protects against transmission corruption.
Compression is global across the image. LZ77’s 32KB sliding window means row 400 can match against row 380 if they’re similar. This is why a 64×64 solid color block compresses to 136 bytes while a 64×64 gradient of the same dimensions takes 10,362 bytes, against 12,288 bytes raw. Uniformity compresses; novelty doesn’t.
And a practical one: for that solid-color block, encoding as indexed color with a one-entry palette produces a 99-byte file instead of 136, because each pixel is one index byte instead of three samples. If your image has few colors, color type 3 usually beats truecolor even after DEFLATE gets its turn.
Interlacing, Briefly
If the interlace byte in
IHDRis 1, the image uses Adam7: the pixels are transmitted in seven passes over an 8×8 grid, coarse to fine, so a partially-downloaded image renders as a low-resolution preview that sharpens.Two things to know. Each pass is filtered and encoded as an independent sub-image with its own scanlines and filter bytes, so a decoder can’t treat the stream as one grid. And Adam7 typically makes files larger, because breaking the image into seven sparse sub-images destroys exactly the local coherence that filtering and LZ77 depend on. It was a good trade on a 28.8k modem. On any modern connection it costs size and complexity for a progressive render nobody waits around to see.
What This Buys You
The design decisions hold up well for a 1996 format:
- Unknown chunks are safe by construction, so the format extended to EXIF metadata, ICC profiles, and animation without ever breaking old decoders.
- Every chunk is individually checksummed, so corruption is localized and detectable rather than silently rendering garbage.
- Filtering is a preprocessing step, not a compression format, which means encoders can get better forever without touching the spec. A file written by
zopflipngtoday decodes fine in a 1997 reader.
That extensibility is not just historical. PNG got a Third Edition as a W3C Recommendation on 24 June 2025, which finally standardized APNG, added an
eXIfchunk for camera metadata, and brought in HDR through three new chunks (cICP,mDCV,cLLI). Thirty years on, the container still had room.Where it shows its age is DEFLATE, which is a 1990s compressor. Lossless WebP does beat it: Google’s own study puts WebP lossless at 23% smaller than PNGs already optimized with ZopfliPNG, and 42% smaller than default libpng output. Worth noting the baseline matters enormously there, and Google’s WebP FAQ quotes a different figure (26%) than the study it links to.
Lossless AVIF is a murkier story than the marketing suggests. AOMedia publishes no general lossless-AVIF-versus-PNG number at all; its quantified claims (50% versus JPEG, 30% versus WebP) are all about lossy encoding. The only primary figure available is 10% versus a 16-bit PNG for a single demo image using a new v1.2.0 feature. Independent testing regularly finds lossless AVIF producing larger files than PNG for flat synthetic images like icons, UI, and charts. If you’re picking a format for screenshots and diagrams, test on your own images rather than trusting a general ranking.
Next in the series: the opposite of all this. A text file, which announces nothing about itself at all.
Sources
- W3C PNG Specification, Third Edition — the current standard; §7.2 covers scanlines and sample packing, §9 covers filtering
- RFC 2083 — the original 1997 IETF PNG specification
- RFC 1950 (zlib) and RFC 1951 (DEFLATE) — the compression layer
- libpng PNG history — the Unisys announcement, the PBF name, and the January 1995 timeline
- WebP Lossless and Alpha Study — Google’s 23%/42% lossless figures and their baselines
- AOMedia on AVIF v1.2.0 — the 10% lossless figure, and its narrow scope
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].