Programming
-
Nobody Agrees What a CSV Is
CSV is simple but powerful. Values, separated by commas. It’s easy to understand and use.
It is also, by a wide margin, the one that destroys the most data.
That’s not a paradox. It’s cause and effect. A format simple enough that everyone writes their own parser is a format with as many dialects as it has parsers, and CSV’s defining property is that it carries no information about how to read it.
There Is No Standard
RFC 4180 exists. Yakov Shafranovich published it in October 2005, it registers the
text/csvmedia type, and it gives an ABNF grammar.It is also Informational, not Standards Track. It was written to describe what people were already doing, twenty-odd years after spreadsheets started emitting it. Section 2 says so outright: there is “no formal specification in existence,” and what follows documents “the format that seems to be followed by most implementations.”
By the time someone wrote it down, every spreadsheet, database, and scripting language had already shipped its own interpretation. The RFC didn’t settle anything. It just added one more dialect, with the distinction of having a number.
Eight years later, RFC 7111 added URI fragments for pointing at a row, column, or cell inside a
text/csvfile. Also Informational. CSV still had no standard, but you could now cite a specific cell of one.
Where Does a Record End?
The obvious answer is “at the newline,” and the obvious answer is wrong, because a quoted field is allowed to contain one.
name,notes Alice,"line one line two" Bob,fineThat’s a valid three-row CSV. Split it on newlines and you get four:
naive split gives 4 lines: 'name,notes' 'Alice,"line one' 'line two"' 'Bob,fine' a real CSV parser gives 3 rows: ['name', 'notes'] ['Alice', 'line one\nline two'] ['Bob', 'fine']Every
head,wc -l,split("\n"), and shell pipeline that assumes one record per line is wrong on this file. Not wrong on a malformed file. Wrong on a correct one.This is the single most common CSV bug, and it’s invisible in testing, because your test fixtures don’t have newlines in them until a user pastes an address into a form.
What’s the Delimiter?
In most of Europe the decimal separator is a comma.
12,50is twelve and a half euros. Which means a comma cannot also be a field separator, so those locales use semicolons.Feed a German CSV to an RFC 4180 parser and everything survives, in the sense that nothing throws:
input: produkt;preis Kaffee;12,50 Tee;9,90 parsed as comma-delimited: ['produkt;preis'] ['Kaffee;12', '50'] ['Tee;9', '90'] parsed as semicolon-delimited: ['produkt', 'preis'] ['Kaffee', '12,50'] ['Tee', '9,90']The first reading gives you two columns of nonsense with no error. The prices split down the middle of the decimal point. A pipeline that ingests this will happily compute statistics on the number 12 and the number 50.
Excel picks the delimiter based on your operating system’s regional settings, which means the same file opens differently on two machines in the same office.
Is the First Row a Header?
1,2,3 4,5,6Header or data? Nothing in the file says.
RFC 4180’s answer is that you put it in the MIME type:
text/csv; header=present. Which is a real answer, and also means the information lives outside the file, in a transport layer that gets stripped the moment someone saves the attachment to disk.So in practice every tool guesses, usually by checking whether the first row looks less numeric than the rest.
The Part That Destroys Data
Everything above is a parsing problem. This one is worse, because the file parses fine and the damage happens after.
CSV has no types. Every value is text. So every spreadsheet and dataframe library applies type inference on import, and type inference is lossy.
Here’s a file with four columns of identifiers, all of which are strings that happen to be made of digits:
gene,zip,card,accession SEPT7,02138,4532012345678901,0004928 MARCH1,01234,4111111111111111,0000071Read it with a type-inferring reader and:
gene zip card accession SEPT7 2138 4532012345678901 4928 MARCH1 1234 4111111111111111 71The ZIP code
02138is now2138. The accession number0004928is now4928. Nobody was asked. Nothing warned. Save that back to CSV and the original values are gone from disk.Spreadsheets are worse than this, because they store every number as an IEEE 754 double. Microsoft is blunt about the consequence:
Excel has a maximum precision of 15 significant digits, which means that for any number containing 16 or more digits, such as a credit card number, any numbers past the 15th digit are rounded down to zero.
The example they reach for is a card number:
typed into a cell 1234 5678 9087 6543 Excel shows 1.23E+15Microsoft calls that “truncating numerical data to 15 digits of precision and converting to a number displayed in scientific notation.” Note that this is the vendor describing its own product, not a bug report.
The card number in the file above is also 16 digits. pandas read it back intact. Excel would not.
Credit card numbers are 16 digits. Many national ID numbers are longer. They are not numbers in any meaningful sense, they’re strings of digits, and a format with no type information cannot tell the difference.
The Gene Name Problem
The best-documented case of this is genomics, because biologists name genes things like
SEPT1andMARCH1and spreadsheets read those as dates.In 2016 Ziemann and colleagues screened 35,175 supplementary Excel files from 18 journals covering 2005 to 2015. Among articles containing Excel gene lists, 19.6% had gene names corrupted this way. One in five.
A follow-up in 2021, “Gene name errors: Lessons not learned,” found 30.9% across a broader sample drawn from PubMed Central. Worth being careful comparing those two numbers directly, because the second study used a different sampling frame and also detected an additional error category the first one didn’t look for. The honest summary is that the problem did not go away in the five years after being loudly published.
The resolution is the remarkable part. The field did not fix the spreadsheets. It renamed the genes. The HUGO Gene Nomenclature Committee’s 2020 guidelines state that “all symbols that auto-converted to dates in Microsoft Excel have been changed,” giving
SEPT1becomingSEPTIN1andMARCH1becomingMARCHF1as examples.Human genes were renamed because a file format cannot say what type a column is.
What To Do About It
CSV isn’t going away, and mostly shouldn’t. It’s readable, streamable, diffable, and every tool on earth reads it.
The practical defenses are short:
- Quote everything. It’s never wrong and it removes a whole class of ambiguity.
- Treat identifiers as strings explicitly at the point of import. Every serious CSV reader lets you pin column types; use it.
- Never round-trip through a spreadsheet if the data contains identifiers. Opening and saving is a lossy operation.
- Say what you mean out of band. Delimiter, encoding, header presence, quoting style. The file will not.
- Use something else when you can. Parquet and even JSON Lines carry types. If the consumer is a program rather than a person, the readability argument for CSV mostly evaporates.
The lesson generalizes past CSV, and it’s the same one from the text file post. A format that carries no description of itself pushes that burden onto every reader, and readers guess. Usually well. Occasionally by silently deleting the leading zero from your ZIP code.
Sources
- RFC 4180 — the Informational spec that documents CSV rather than defining it
- RFC 7111 — URI fragment selectors for
text/csv, January 2014 - Ziemann et al., “Gene name errors are widespread in the scientific literature” — Genome Biology, 2016; the 19.6% figure (free full text)
- Abeysooriya et al., “Gene name errors: Lessons not learned” — PLOS Computational Biology, 2021; the 30.9% follow-up
- Bruford et al., “Guidelines for human gene nomenclature” — Nature Genetics, 2020; the renaming (free full text)
- Microsoft on Excel’s floating-point precision — Excel follows IEEE 754 and stores 15 digits of precision
- Microsoft on leading zeros and large numbers — “any numbers past the 15th digit are rounded down to zero,” with a credit card as the example
- Microsoft on importing and exporting text files — the CSV list separator comes from Windows Region settings
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Word Documents Used to Be Filesystems
Last post ended on a promise: a
.docxis a ZIP file, and you already know how ZIP works.That’s true, and it’s the smaller half of the story. The interesting part is what
.docxreplaced, because the old.docformat was doing something strange. It wasn’t a document. It was a filesystem with a document living inside it.
A Filesystem in a File
Here’s a real
.docfrom 2015, 10,240 bytes. The first eight bytes:d0 cf 11 e0 a1 b1 1a e1That’s the Compound File Binary Format signature, also called OLE2.
file(1)recognizes it and doesn’t even mention Word:$ file "rich text.doc" Composite Document File V2 Document, Little Endian, Os: Windows, Version 1.0, Code page: -535, Revision Number: 0, Create Time/Date: Thu Dec 10 13:38:22 2015“Composite Document File” is the honest description. CFBF is a container that implements directories and files, called storages and streams, inside a single flat file. It has a File Allocation Table. It has sectors. If that sounds like FAT16, that’s because it’s the same idea, scaled down to live inside one file on a real filesystem.
Cracking this one open gives:
106 bytes CompObj 20 bytes Ole 116 bytes DocumentSummaryInformation 312 bytes SummaryInformation 2411 bytes 1Table 3620 bytes WordDocument sector size : 2^9 = 512 bytes mini sector size : 2^6 = 64 bytesTwo sector sizes, because a 512-byte sector is wasteful for a 20-byte stream. Streams under 4,096 bytes get allocated out of a separate mini-FAT in 64-byte units. There is a fragmentation strategy inside your Word document.
The
WordDocumentstream is the main event, and it opens with a File Information Block whose magic number is0xA5EC:WordDocument stream: 3620 bytes FIB magic (wIdent) = 0xA5ECNone of this is the text yet. This is all container.
The Text Is Not in Order
You’d expect the document’s text to sit in the
WordDocumentstream in reading order. It doesn’t. It sits there in edit order, and a separate structure called a piece table says how to reassemble it.The piece table is a list of descriptors, each saying “characters at logical position X through Y live at physical offset Z.” Reading a
.docmeans walking that table and gathering fragments scattered through the stream.Why build it that way? Because of a feature called Fast Save, and because in 1990 writing to disk was slow. When you edited a document, Word didn’t rewrite the file. It appended your new text to the end of the stream and updated the piece table to point at it. Saving a one-word change to a 200-page document meant writing a few dozen bytes instead of a few hundred kilobytes.
That’s a good optimization. It has an obvious and terrible consequence.
The old text is still in the file. Deleting a paragraph removed it from the piece table, not from the stream. The bytes stayed exactly where they were, unreferenced, invisible in Word, and completely readable in a hex editor.
Microsoft documented this themselves, in a knowledge base article about minimizing metadata in Word documents: “Because of the design of the FastSave feature, text that you delete from a document may remain in the document, even after you save the document.” The recommended fix was to go into Options and clear the “Allow fast saves” check box. From Word 97 SR-1 onward they turned it off by default.
For years, “open the document in a text editor and scroll” was a functioning technique for reading text someone believed they had deleted. Every organization circulating Word files was potentially shipping its own edit history.
The piece table itself has a respectable pedigree. Charles Simonyi brought the technique to Microsoft from Xerox PARC’s Bravo editor, and it’s an elegant way to represent an editable buffer. It’s still how many text editors model documents in memory. The mistake wasn’t the data structure. The mistake was persisting the whole scratch buffer to disk and shipping it to other people.
Then It Became a ZIP of XML
Office 2007 replaced all of it with the Open Packaging Conventions: ECMA-376, later ISO/IEC 29500. A
.docxis a ZIP archive containing XML.Every
.docxopens with the same four bytes:50 4b 03 04 <- PK\x03\x04, a ZIP local file headerPK. Phil Katz’s initials, from the last post, sitting at byte zero of every Word document written since 2007.Unzip one and the structure is legible:
[Content_Types].xml _rels/.rels word/document.xml word/_rels/document.xml.rels word/styles.xml word/settings.xml word/fontTable.xml word/theme/theme1.xml docProps/core.xml docProps/app.xmlword/document.xmlholds the text.[Content_Types].xmlmaps each part to a MIME type._rels/.relsis a relationship graph saying which part is the main document and how the parts connect. The whole thing is a tiny website, zipped.The text itself is WordprocessingML:
<w:p> <w:r> <w:t>Hello, World!</w:t> </w:r> </w:p>A paragraph containing a run containing text. Verbose, but you can read it, and more importantly a program you wrote in an afternoon can read it. That is important when building foundational file formats that outlive the creators.
Extracting text from a
.docmeant implementing a filesystem and a piece table. Extracting text from a.docxmeans unzipping and finding<w:t>elements.The XML contains the document, not the document’s history. Deleted text is deleted.
XML Did Not Mean Simple
It would be tidy to end on “and then it got clean.” The specification runs to several thousand pages, and the ISO fast-track that pushed it through in 2008 was contentious enough to deserve its own post.
What matters here is the shape it settled into. The standard shipped split in two: Strict, the clean format, and Transitional, which carries the legacy baggage forward so documents converted from the binary era still render correctly.
Guess which one nearly everything emits.
Open a Transitional document’s settings and you find a
<w:compat>block. Its children are a museum:w:truncateFontHeightsLikeWP6 WordPerfect 6 w:suppressTopSpacingWP WordPerfect w:lineWrapLikeWord6 Word 6 w:autoSpaceLikeWord95 Word 95 w:footnoteLayoutLikeWW8 Word 97 w:useWord97LineBreakRules Word 97 w:mwSmallCaps Mac WordEvery one of those is a flag asking the renderer to reproduce how a specific piece of 1990s software behaved. Not what the format should do. What Word 6 did do, quirks included. Implementing this correctly means emulating applications whose behavior was never written down anywhere.
The bugs were load-bearing, so they got standardized. The format stopped being a filesystem, but it did not stop being a thirty-year-old application’s memory dumped to disk. It just picked a more legible way to write it down.
Which is, in fairness, an enormous improvement. You can read the file now. You just can’t read all of it quickly.
Sources
- MS-CFB: Compound File Binary Format — Microsoft’s spec for the OLE2 container
- MS-DOC: Word Binary File Format — the FIB, the piece table, and the stream layout
- ECMA-376 — Office Open XML, the basis for
.docx, and free to download. This is the same specification ISO published as ISO/IEC 29500, so read it here rather than paying ISO for the identical text - Library of Congress format description for OOXML — preservation notes and format history
- KB Q223790: WD97: How to Minimize Metadata in Word Documents — the fast-save warning, archived; Microsoft no longer hosts it
w:compatschema reference — the full list of compatibility settings, browsable without downloading the spec
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
ZIP Files Are Read Backwards
Every format in this series so far reads front to back. PNG starts with a signature and you walk chunks in order. A text file is bytes from the beginning. Markdown parsers scan line by line, top to bottom.
ZIP reads backwards. The index is at the end of the file, and a reader is expected to seek to the end first and work its way back.
That one decision explains almost everything strange about ZIP, including a few things that look like bugs and one thing that is definitely a bug.
The Index Lives at the End
Here’s a real ZIP containing two small text files. 241 bytes total. Scanning it for the four-byte record signatures gives the whole layout:
offset bytes ascii record 0 50 4b 03 04 P K . . Local File Header <- hello.txt 53 50 4b 03 04 P K . . Local File Header <- second.txt 108 50 4b 01 02 P K . . Central Directory Header 163 50 4b 01 02 P K . . Central Directory Header 219 50 4b 05 06 P K . . End of Central DirectoryEvery one of those starts with the same two bytes.
0x50is decimal 80, which isPin ASCII.0x4bis decimal 75, which isK.PK. Phil Katz, who wrote PKZIP in 1989, put his initials in the first two bytes of every structure in the format, and they are still there in every.docx,.jar, and.epubon your machine.The two bytes after
PKare the record type:03 04for a local file header,01 02for a central directory entry,05 06for the end-of-central-directory record. Those aren’t printable characters, which is deliberate. A four-byte constant made of two readable letters and two control bytes is unlikely to appear by accident in text, and easy to spot by eye in a hex dump.The last 22 bytes are the End of Central Directory record, and it’s the entry point:
50 4b 05 06 00 00 00 00 02 00 02 00 6f 00 00 00 6c 00 00 00 00 00 signature 0x06054b50 total CD records 2 central dir size 111 bytes central dir offset 108 comment length 0A reader opens the file, jumps to the end, finds that record, reads “the index is at offset 108,” seeks there, and reads the catalog. Listing the contents of a 4 GB archive touches a few hundred bytes.
Note that each file appears twice: once as a Local File Header immediately before its compressed data, and once as an entry in the Central Directory at the end. Hold that thought.
Why Would You Do This?
Because in 1989 you were writing to a floppy disk, and often to a floppy disk that wasn’t big enough.
If the index goes at the front, you have to know everything about every file before you write the first byte: how many files, how big each one compresses to, where each one lands. That means compressing everything to a temporary location, then writing the header, then copying it all back. On a machine with 640K of RAM and two floppy drives, that’s brutal.
Put the index at the end and you can stream. Compress a file, write it, remember where it went. Compress the next one. When you run out of files, write down everything you remembered. One pass, no temporary copy, and you never needed to know the total size in advance.
TAR solved the same problem by having no index at all, which is why
tarhas to read an entire archive to find one file, and why you cannot randomly access a.tar.gz. ZIP got both streaming writes and random-access reads. That’s the trade that made it win.
The Backwards Scan Is Fuzzier Than It Sounds
The EOCD record is 22 bytes, so you’d think a reader could just read the last 22 bytes and be done.
It can’t, because the record ends with a variable-length archive comment of up to 65,535 bytes. The signature isn’t at a fixed offset from the end of the file. So a reader has to seek near the end and scan backwards looking for the four-byte signature, potentially across 65,557 bytes.
Searching for a magic number is not the same as knowing where a structure is. If those four bytes happen to appear inside the comment, or inside compressed data near the end of the file, a naive parser can lock onto the wrong one. Different implementations pick different candidates when there’s more than one. This is a recurring source of “this archive opens in one tool and not another.”
You Can Put Anything in Front of a ZIP
If a reader finds the archive by scanning backwards from the end, then whatever sits at the front of the file is not the reader’s problem.
Take a valid 69-byte PNG, take the 241-byte ZIP, and concatenate them with
cat. No special tooling:$ file polyglot.png polyglot.png: PNG image data, 1 x 1, 8-bit/color RGB, non-interlaced $ unzip -l polyglot.png Length Date Time Name --------- ---------- ----- ---- 12 08-10-2026 16:44 hello.txt 13 08-10-2026 16:44 second.txt --------- ------- 25 2 filesOne 310-byte file. An image viewer reads the PNG signature at byte 0 and renders an image. An archive tool scans backwards, finds the EOCD, and extracts two files. Both are correct. Neither is being fooled by a trick; they’re each doing exactly what their format says to do.
This is the mechanism behind self-extracting archives, where the front of the file is a real executable and the back is a real ZIP. The same property is why “GIFAR” attacks worked: a file that a server accepted as a harmless image was loaded by Java as an archive of classes.
It also means the offsets inside the Central Directory are relative to the start of the archive, not the start of the file, and readers have to work out that difference. Prepending data shifts everything, and well-behaved parsers cope by computing the delta between where the EOCD says the directory should be and where it found it.
Two Indexes, One File
Back to that detail from earlier: every file’s name and metadata are stored twice, in the Local File Header and again in the Central Directory.
Nothing enforces that they agree.
Here’s the same archive with only the Central Directory copy of the first filename patched from
hello.txttoBOGUS.txt. The local header is untouched:$ unzip -l mismatch.zip Length Date Time Name --------- ---------- ----- ---- 12 08-10-2026 16:44 BOGUS.txt 13 08-10-2026 16:44 second.txt local file header at offset 0 still says: hello.txtThe archive is not corrupt.
unziplists it happily. It just contains two different answers to “what is this file called,” and which one you get depends on which structure your parser decided to trust.Now imagine two programs reading the same archive, one checking a signature and the other extracting files. That’s the Android “Master Key” bug from 2013, and the detail is better than the summary.
An APK is a ZIP. The attacker puts two entries in it, both named
classes.dex. Android’s Java verifier loaded entries into a map keyed by filename, so a duplicate name overwrote the earlier one and the last entry was the one whose signature got checked. The native installer used a hash table with linear probing that didn’t replace on collision, so the first entry was the one that got loaded and run. Plant malicious code first, legitimately signed code second, and the device verifies one file and executes the other.A second bug the same year came from the same “two readings, one file” family, via a signed integer. The extra-field length is a 16-bit value, and the Java code read it signed. A length of 65,533 (
0xFFFD) sign-extends to −3. Since the offset of the compressed data is computed by adding that length, a negative value moves the read pointer backward into the header region instead of forward past it.The lesson generalizes past ZIP. Any format that stores the same fact twice has to decide what happens when the copies disagree, and “the spec doesn’t say” is the same answer as “attackers decide.”
Offsets Are Just Numbers
The Central Directory locates each file by offset. Nothing in the format says two entries can’t point at the same bytes.
The classic zip bomb didn’t need that.
42.zipis 42 kilobytes of archives nested five layers deep, sixteen at each layer, unpacking to roughly 4.5 petabytes. The defense is obvious once you’ve seen it: cap recursion depth, don’t auto-extract nested archives.David Fifield’s 2019 construction doesn’t recurse at all. It expands in a single pass, so depth limits are irrelevant. The trick is overlap: many Central Directory entries reference one shared kernel of compressed data, and each entry’s compressed stream uses DEFLATE’s stored-block mode to quote the next entry’s local file header as literal bytes. Entries nest inside each other, and output grows quadratically against input.
He published several, and they aren’t interchangeable:
File Compressed Uncompressed Ratio Needs Zip64 zbsm.zip42 KB 5.5 GB ~130,000:1 No zblg.zip10 MB 281.4 TB ~28,000,000:1 No zbxl.zip46 MB 4.5 PB ~98,000,000:1 Yes The Zip64 requirement on the largest one matters, because not every reader supports Zip64, which makes the merely-enormous version the more portable weapon.
None of these are malformed files. Every one is a valid archive that a conforming parser is supposed to accept. The format allows two entries to describe the same bytes, and no rule anywhere says the total uncompressed size has to bear any relationship to the file you’re holding.
Everything Is Secretly a ZIP
Once you know the structure, you start recognizing it:
.docx,.xlsx,.pptxare ZIP archives of XML.jar,.war,.apkare ZIP archives of class files and resources.epubis a ZIP of XHTML.odt,.odsare ZIP of XML again
That’s not a coincidence or a hack. ISO/IEC 21320-1, “Document Container File,” defines a constrained ZIP profile for exactly this use. It narrows the format so a
.docxreader doesn’t have to implement all of ZIP’s accumulated history: compression must be stored or deflated and nothing else, and the various encryption and digital-signature mechanisms in the original spec are all forbidden.It’s a narrowing, not a rewrite. Zip64 version 1 is still permitted, for instance; only version 2 is ruled out. The profile is best understood as a list of the parts of ZIP that turned out to be a bad idea.
Which means the next post in this series is mostly about a ZIP file with XML inside it. You already know half of how a Word document works.
Sources
- PKWARE APPNOTE.TXT — the original and still-authoritative ZIP specification, currently version 6.3.10; §4.3.16 defines the end of central directory record
- ISO/IEC 21320-1:2015 — the constrained ZIP profile used by document formats. Fair warning, this one is a paid ISO standard; the catalog page tells you what it covers but you cannot read the text without buying it
- Library of Congress format description for ZIP — history and preservation notes
- David Fifield: A Better Zip Bomb — the overlapping-stream construction
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Markdown Is Not a Format, It's an Argument
I’ve covered PNG and text files, and now it’s time for Markdown, which can be thought of as a philosophy of formatting or a lifestyle of text documents more so than an actually well defined file format. It has structure, and it has specifications, plural, and nothing agrees.
Here is three lines of Markdown run through five parsers:
INPUT: "- outer\n - inner\n" Python-Markdown <ul> <li>outer</li> <li>inner</li> </ul> markdown2 <ul> <li>outer <ul> <li>inner</li> </ul></li> </ul> mistune <ul> <li>outer<ul> <li>inner</li> </ul> </li> </ul> marko (CommonMark) <ul> <li> outer<ul> <li>inner</li> </ul> </li> </ul> cmark-gfm (GitHub) <ul> <li>outer <ul> <li>inner</li> </ul> </li> </ul>Five parsers, five different results. Most of that is cosmetic whitespace, but look at the first one: Python-Markdown produced a flat list. The nesting is gone. That’s not a formatting difference, that’s a different document.
The Original Spec Was an Essay
John Gruber released Markdown in March 2004, along with a Perl script called
Markdown.pl. The design goal was stated plainly:The overriding design goal for Markdown’s formatting syntax is to make it as readable as possible. The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions.
That goal was met, and it’s why we’re all still using it twenty years later. The syntax borrowed from conventions people had already invented for plain text email and Usenet:
=and-underlines from Setext,#headers from atx,>quoting from Usenet,*for emphasis from Textile and reStructuredText. None of it was new. That was the point.What Markdown shipped without was a grammar. The specification was English prose describing the syntax with examples, and the tiebreaker for anything the prose didn’t cover was “whatever
Markdown.pldoes.” A Perl script full of regular expressions became the definition of the format by default.That works fine until someone writes a second implementation.
Where the Prose Ran Out
The ambiguities weren’t exotic. They were things you hit in the first week:
How much indentation nests a list? Two spaces? Four? One tab? The original prose didn’t say clearly, and the answer interacts with the rule that four spaces means a code block.
What happens inside raw HTML? If you write a
<div>and put Markdown inside it, does the Markdown get processed? Gruber’s implementation had behavior; the prose didn’t specify it.When does a
*open emphasis versus just being an asterisk? Ina * b * c, are those multiplication signs or emphasis delimiters?Do underscores work inside words? This one bites daily:
INPUT: "snake_case_variable" Python-Markdown <p>snake_case_variable</p> markdown2 <p>snake<em>case</em>variable</p> mistune <p>snake_case_variable</p> marko (CommonMark) <p>snake_case_variable</p> cmark-gfm (GitHub) <p>snake_case_variable</p>markdown2 italicizes your variable name. Every other parser leaves it alone. Both are defensible readings of a spec that never addressed it.
Or the heading with no space after the hash:
INPUT: "#Heading" Python-Markdown <h1>Heading</h1> markdown2 <h1>Heading</h1> mistune <p>#Heading</p> marko (CommonMark) <p>#Heading</p> cmark-gfm (GitHub) <p>#Heading</p>Half of them give you a heading, half give you a paragraph starting with a hash. This one matters because
#hashtagat the start of a line is a real thing people write.
Everyone Wrote Their Own
With no formal spec, every implementation became a dialect, and the popular ones added features:
- PHP Markdown Extra (Michel Fortin, 2005) added pipe tables, definition lists, footnotes, fenced code blocks, and attribute blocks.
- MultiMarkdown (Fletcher Penney, 2005) added metadata frontmatter, cross-references, citations, and LaTeX export.
- Pandoc Markdown (John MacFarlane, 2006) built a real AST-based parser and added YAML frontmatter, TeX math, grid tables, and citations.
- kramdown (Thomas Leitner, 2009) added inline attribute lists and its own math support.
Each is a superset of a slightly different reading of the original. A document written for one is not guaranteed to render correctly in another, and the failure mode is silent: you don’t get a parse error, you get the wrong document.
CommonMark: Specify the Ambiguity Away
On 3 September 2014, Jeff Atwood announced a spec effort on Coding Horror under the name Standard Markdown, with John MacFarlane as primary author and people from GitHub, Reddit, Stack Exchange, and Meteor involved. The goal was not a new dialect and not a replacement for Gruber’s syntax, but an unambiguous description of what the existing syntax should mean in every case.
The name lasted about a day. That night, by Atwood’s account, Gruber emailed him and MacFarlane privately, called the name “infuriating,” and asked that the project be renamed and the domain taken down. On 4 September, Atwood published a follow-up retitling it Common Markdown, which shortly became the one-word CommonMark.
Worth being precise here, because this story gets retold badly: this was not a trademark action. Gruber holds no registered trademark on “Markdown” and did not invoke one. It was an objection to the name, made in private email, and the only public record of his side is Atwood’s paraphrase. There is no Daring Fireball post about it.
The naming fight is a footnote. The approach is the interesting part. Rather than describing the syntax in prose and hoping, CommonMark defines a parsing algorithm and ships an executable test suite pairing exact input with exact expected HTML, more than 500 examples embedded in the spec document itself. Conformance is not a matter of opinion. You run the tests.
The algorithm works in two passes.
Phase one walks the document line by line and builds block structure. Container blocks (blockquotes, lists, list items) and leaf blocks (headings, code blocks, paragraphs, HTML blocks) get assembled into a tree. Link reference definitions get collected. No inline formatting is considered at all in this phase, which is why block structure always wins: a
>at the start of a line is a blockquote marker regardless of what emphasis you thought you were in the middle of.Phase two walks the text inside leaf blocks and resolves inline structure. This is where emphasis, links, images, code spans, and inline HTML get parsed, using a delimiter stack.
That two-phase split is the single most useful thing to know about Markdown parsing, because it explains most surprising behavior. If your emphasis “leaked” across a list item boundary, it didn’t; blocks were decided before emphasis was ever considered.
The Emphasis Rules Are Hard
Emphasis is the hardest part of the spec, and CommonMark’s solution is a set of flanking rules. A run of
*or_is classified as left-flanking (can open emphasis) or right-flanking (can close it) based on the characters on either side, roughly: a delimiter can open if it’s not followed by whitespace, and can close if it’s not preceded by whitespace, with extra conditions around punctuation.Then there’s a special case for underscores: an
_can open emphasis only if it’s left-flanking and not right-flanking. That single asymmetry is what makessnake_case_variablesafe, because the middle underscores are both left- and right-flanking and are therefore disqualified from opening anything. Asterisks don’t get that rule, which is whysnake*case*variablestill italicizes.This is what “specifying the ambiguity away” costs. The rule isn’t elegant. It exists because real documents contain identifiers, and a spec that italicizes your variable names is wrong no matter how clean its grammar is.
You can see the payoff in the nesting case:
INPUT: "*foo**bar**baz*" Python-Markdown <p><em>foo</em><em>bar</em><em>baz</em></p> everyone else <p><em>foo<strong>bar</strong>baz</em></p>Four parsers agree, and the one that predates the delimiter-stack approach gets it wrong in a way that changes the meaning.
GFM Is a Layer, Not a Fork
GitHub Flavored Markdown is CommonMark plus five extensions, and it’s specified against CommonMark rather than diverging from it:
- Tables, pipe-delimited with alignment colons
- Task lists,
- [ ]and- [x], rendered as checkboxes - Strikethrough,
~~text~~ - Autolinks, bare URLs linkified without brackets
- A raw HTML filter that neutralizes dangerous tags by escaping their opening bracket
That last one is a security control rather than a formatting feature, which tells you something about what it’s like to run a Markdown renderer on user-submitted content at GitHub’s scale.
The extension boundary is visible if you feed the same table to both:
INPUT: | a | b | |---|---| | 1 | 2 | CommonMark <p>| a | b | |---|---| | 1 | 2 |</p> cmark-gfm <table><thead><tr><th>a</th><th>b</th></tr></thead>...Tables are not Markdown. Tables are a GFM extension. CommonMark renders that input as a paragraph containing literal pipe characters, and it is correct to do so.
Tables, footnotes, task lists, strikethrough, frontmatter, math, and Mermaid diagrams are all extensions. None of them are guaranteed anywhere.
What To Do About It
The practical takeaways are short.
Know which parser you’re targeting. “It renders on GitHub” tells you about cmark-gfm, and nothing about your static site generator, your docs pipeline, or someone’s RSS reader.
Prefer the constructs everyone agrees on. Headings with a space after the hash, fenced code blocks, asterisks for emphasis, blank lines between blocks, four-space or consistent nesting. Boring Markdown survives transport.
Don’t rely on parser-specific behavior you discovered by accident. If nesting a list at two spaces works in your tool, that’s your tool, not the format.
There is even a formal way to say which dialect you mean. RFC 7763 registers
text/markdownas a media type, and RFC 7764 defines avariantparameter for exactly this problem:text/markdown; variant=CommonMark text/markdown; variant=GFM text/markdown; variant=OriginalThe standards process looked at Markdown, concluded that saying “this is Markdown” is not specific enough to be useful, and standardized a way to say which Markdown you meant.
That’s the tradeoff Markdown made. PNG picked one answer and enforced it with a checksum. A text file refuses to answer anything. Markdown let a million answers bloom, got adopted everywhere precisely because it was easy to implement badly, and has spent the last decade trying to agree with itself.
I’ll take that trade. But it’s worth knowing that when you write Markdown, you are not writing in a format. You’re writing in a dialect, and hoping the reader speaks it.
Sources
- Daring Fireball: Markdown — Gruber’s original 2004 syntax document and design goals
- CommonMark Specification — the parsing algorithm, emphasis flanking rules, and executable test suite
- CommonMark parsing strategy appendix — the two-phase block/inline design
- GitHub Flavored Markdown Spec — the five extensions, specified against CommonMark
- RFC 7763 and RFC 7764 — the
text/markdownmedia type and the registered dialect variants, both by S. Leonard, March 2016 - Coding Horror: Standard Flavored Markdown and Standard Markdown is now Common Markdown — Atwood’s announcement and the rename a day later
- Daring Fireball: Introducing Markdown — the original 15 March 2004 announcement
tagfilter.cin cmark-gfm — the nine tags GFM’s raw HTML filter neutralizes
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
There Is No Such Thing as a Text File
Last time I took apart PNG, which opens with eight bytes whose entire job is to announce “I am a PNG”.
A text file opens with nothing. No signature, no header, no length field, no version, no metadata. It is bytes, and then it stops.
So this post is the opposite of the last one. Instead of walking a structure, we’re going to look at what happens when there isn’t one.
POSIX Defines the Standard
Start with the standard, in §3.403:
A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the
<newline>character. Although POSIX.1-2017 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files.The standard defines the term and then tells you the system doesn’t enforce it.
Nothing in the filesystem records “this is text.” There’s no flag on the inode, no attribute, nothing in the directory entry. The
.txtextension is a hint to humans and to Windows. “Text file” is not a property a file has. It’s a claim the reader makes about the bytes, and every tool makes it slightly differently.
The Bytes Don’t Know What They Mean
A file stores bytes. Turning bytes into characters requires an encoding, and the encoding is not in the file.
Here are the same five characters,
Héllo, in several encodings:ascii FAILS: ordinal not in range(128) latin-1 5 bytes 48 e9 6c 6c 6f cp1252 5 bytes 48 e9 6c 6c 6f utf-8 6 bytes 48 c3 a9 6c 6c 6f utf-16 12 bytes ff fe 48 00 e9 00 6c 00 6c 00 6f 00 utf-16-be 10 bytes 00 48 00 e9 00 6c 00 6c 00 6f utf-32 24 bytes ff fe 00 00 48 00 00 00 e9 00 00 00 ...Five characters. Anywhere from 5 to 24 bytes. Nothing in any of those files says which one you’re looking at.
When you open a file in your editor and it looks right, that’s your editor guessing correctly. When you get
caf√©instead ofcafé, that’s your editor guessing wrong. The file never changed.
The Fight Over the Eighth Bit
ASCII was standardized as a 7-bit code: 128 values,
0x00through0x7F. Thirty-three control codes, ninety-five printable characters, and that was the whole world if the world spoke English.Bytes have eight bits though, so there were another 128 values sitting there unused. Everyone grabbed them, and everyone grabbed them differently.
ISO 8859-1 (Latin-1) claimed
0xA0–0xFFfor Western European letters and reserved0x80–0x9Ffor a second set of control codes nobody used. Microsoft looked at those 32 wasted slots and put printable punctuation there instead, creating Windows-1252. That’s where the curly quotes and the em dash live:CP1252 text : It's "fine" — really CP1252 bytes : 49 74 27 73 20 93 66 69 6e 65 94 20 97 20 72 65 61 6c 6c 79 UTF-8 bytes : 49 74 27 73 20 e2 80 9c 66 69 6e 65 e2 80 9d 20 e2 80 94 ...Byte
0x93is a left curly quote in CP1252 and a control character in strict Latin-1. This is why pasting from Word into a system expecting Latin-1 produces garbage: the bytes are legal, they just mean nothing there.Mojibake, Japanese for “character transformation,” is exactly this, and it’s completely deterministic:
original text : café as UTF-8 bytes : 63 61 66 c3 a9 read as CP1252 : caféc3 a9is one character in UTF-8 and two characters in CP1252. Both readings are valid. Only one is what you meant.It could have been worse. IBM’s EBCDIC, still running on mainframes, isn’t an ASCII superset at all:
'A' ASCII 0x41 EBCDIC 0xc1 'a' ASCII 0x61 EBCDIC 0x81 ' ' ASCII 0x20 EBCDIC 0x40And the letters aren’t even contiguous.
Iis0xc9,Jis0xd1, with a gap in between. Sorting strings by byte value, which works fine in ASCII, silently produces wrong output in EBCDIC.
Why UTF-8 Won
UTF-8 encodes a character in one to four bytes. ASCII characters keep their single-byte values, so every ASCII file is already a valid UTF-8 file. That backward compatibility gets most of the credit, but the more interesting property is the bit pattern:
'A' U+0041 1 byte 41 01000001 'é' U+00E9 2 bytes c3 a9 11000011 10101001 '€' U+20AC 3 bytes e2 82 ac 11100010 10000010 10101100 '🙂' U+1F642 4 bytes f0 9f 99 82 11110000 10011111 10011001 10000010Look at the leading bits. A single-byte character starts with
0. A multi-byte character starts with110,1110, or11110, where the number of leading 1s is the total byte count. Every continuation byte starts with10, and nothing else does.That makes UTF-8 self-synchronizing. Drop into the middle of a file at a random offset and you can tell immediately whether you’re mid-character, and walk backwards a byte or two to find the boundary. You do not need to have read the file from the beginning.
Compare that to UTF-16, where you must know the byte order and must have tracked whether you’re on an even or odd boundary. UTF-8 made encoding a local property instead of a global one, and that’s why it took over.
The BOM
Multi-byte encodings have a byte order problem: is
00 48the characterU+0048orU+4800? The Byte Order Mark solves it by puttingU+FEFFat the start of the file, so a reader can look at the first two bytes and work out the endianness.utf-16 ff fe 68 69 ... (little-endian) utf-16-le 68 00 69 00 (no BOM, you'd better know) utf-8-sig ef bb bf 68 69 (UTF-8 "BOM") utf-8 68 69 (no BOM)UTF-8 has no byte order to mark, because its unit is one byte. The UTF-8 BOM is not a byte order mark at all; it’s a three-byte flag saying “this is UTF-8,” and the Unicode Consortium neither requires nor recommends it.
It also actively breaks things. The kernel identifies a script by looking for
0x23 0x21, the characters#!, at offset zero:no BOM : 23 21 2f 62 69 6e 2f 73 68 0a -> #!/bin/sh with BOM: ef bb bf 23 21 2f 62 69 6e 2f -> not a scriptSame for JSON parsers, CSV importers, and anything else that expects a specific first byte. If you have ever seen a shell script fail with a cryptic error on a line that looks correct, this is a candidate.
Lines Are a Convention Too
There is no line structure in a text file. There’s a byte that tools agree means “line break,” and even that isn’t agreed on.
The split is a hardware inheritance. A teletype needed two separate mechanical actions to start a new line: carriage return (
0x0D) moved the print head back to the left margin, and line feed (0x0A) advanced the paper by one row. Two actions, two control codes.Then everyone picked differently. Unix chose LF alone. MS-DOS, and Windows after it, kept both as CRLF. Classic Mac OS used CR alone. Those choices are still with us thirty years later, and they’re the reason
.gitattributesexists.And then there’s the trailing newline, which people argue about without realizing the standard already answered it. POSIX §3.206:
A line is a sequence of zero or more non-
<newline>characters plus a terminating<newline>character.The newline is part of the line, not a separator between lines. A file whose last byte isn’t a newline doesn’t have a final line. POSIX §3.195 has a name for what it has instead: an incomplete line.
That definition has teeth:
$ wc -l lf.txt nofinal.txt 2 lf.txt 1 nofinal.txtBoth files contain the text
oneandtwo. The first ends with a newline, the second doesn’t.wc -lcounts newline bytes, so the second file reports one line despite visibly having two.This is also what git’s
\ No newline at end of filemarker means. It isn’t a style complaint. Git is telling you the last line is incomplete by the POSIX definition, which matters because otherwise appending a line would silently modify the existing last line rather than adding a new one.
How Tools Guess
Since nothing declares itself, every tool that needs to know applies a heuristic. The dominant one is: does it contain a NUL byte?
That test exists because C strings are NUL-terminated, so a NUL in the middle of what claims to be text means something is off. It’s a good heuristic. It’s also wrong in two ways worth knowing about.
Git’s version is
buffer_is_binary()inxdiff-interface.c, and it doesn’t scan the whole file. It caps at a constant:#define FIRST_FEW_BYTES 8000So the check is “is there a NUL in the first 8000 bytes.” A file with clean text for 10KB and a NUL after that is text as far as git is concerned. The cutoff is a performance tradeoff, and it means binary-ness is decided by a sample, not a proof.
The second problem is bigger.
plain ascii NUL present: False -> text utf-8 with emoji NUL present: False -> text has a NUL byte NUL present: True -> BINARY utf-16 text NUL present: True -> BINARYUTF-16 encodes ASCII characters as the character byte plus a NUL. Any UTF-16 file that’s mostly English is roughly half NUL bytes. So git does this to a perfectly valid text file:
$ git diff --cached --stat lf.txt | 2 ++ utf16.txt | Bin 0 -> 24 bytesBin. Git will not diff it, will not merge it, and will not show it in review. The heuristic isn’t detecting text, it’s detecting C-string-safety, and those aren’t the same question.file(1)is more thorough, and it shows how much the BOM is doing:$ file utf16.txt utf16_bom.txt lf.txt utf16.txt: data utf16_bom.txt: Unicode text, UTF-16, little-endian text lf.txt: ASCII textIdentical text content in the first two files. The only difference is two leading bytes. Without them
filegives up and calls itdata; with them it identifies the encoding exactly. For a format with no header, a BOM is the closest thing to one that exists.Under the hood
fileis doing real work rather than one heuristic.src/encoding.ccarries a 256-entry table classifying every byte value as never-valid-in-text, ASCII, ISO-8859, or extended ASCII, plus a dedicated UTF-8 state machine that rejects invalid sequences. It then tries candidate encodings in order: ASCII, UTF-7, UTF-8 with BOM, UTF-8, UTF-32, UTF-16, Latin-1, extended ASCII, and finally EBCDIC. That ordering is a nice fossil record of which encodings are still worth guessing first.
Why This Matters
Nearly every format developers work in daily is a convention layered on this substrate. Source code, JSON, YAML, TOML, CSV, Markdown, config files, logs. All of them inherit these problems, and none of them can fully escape them, because the layer underneath has no way to describe itself.
That’s the tradeoff. A format with no header can’t tell you anything about itself, which is exactly why it has outlived every format that could. PNG will be readable as long as someone maintains a PNG decoder. A text file is readable as long as someone remembers what bytes are.
Next in the series: Markdown, which is a text file plus a set of conventions that nobody fully agrees on.
Sources
- POSIX.1-2017 Base Definitions, Chapter 3 — §3.206 Line, §3.195 Incomplete Line, §3.403 Text File
- RFC 3629 — the UTF-8 specification and its byte patterns
- RFC 2046 §4.1 — the
text/plainmedia type - Unicode FAQ on UTF-8, UTF-16, and the BOM — the Consortium’s own guidance on why not to use a UTF-8 BOM
buffer_is_binary()in git’sxdiff-interface.c— the NUL check and theFIRST_FEW_BYTEScutoffsrc/encoding.cin thefileproject — the text-character table and encoding-guessing order behindfile(1)
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
How PNG Actually Stores Your Pixels
I’m starting a series on file formats. Not “here are the ten image formats you should know,” but the actual bytes: what’s in the file, in what order, and why someone decided it should be that way.
Starting with PNG, because it’s the format most developers touch every day and almost nobody has looked inside.
I am likely to cover a few things that other explainer documents have covered, such as chunk structure and chunk types. However, I’d like to dig into some details that are not often mentioned, such as where your pixels went.
A Format Born From a Patent Fight
PNG exists because of a licensing ambush. On 28 December 1994, right in the middle of the holidays, Unisys announced an agreement to start collecting royalties from authors of GIF-supporting software, on the strength of its patent on the LZW compression algorithm that GIF used.
The response was fast. A draft for a replacement format was posted to
comp.graphicson 4 January 1995, one week later. It was originally called PBF, for Portable Bitmap Format, and got renamed to PNG two days after that. The format shipped as a W3C Recommendation in October 1996.Two things about that origin still show in the bytes. The format is aggressively defensive, because it was designed by people who expected files to be mangled in transit. And it is aggressively extensible, because they had just watched a format become unusable for reasons that had nothing to do with its technical design.
Eight Bytes of Paranoia
Every PNG starts with the same eight bytes:
Hexadecimal: 89 50 4E 47 0D 0A 1A 0A ASCII/Ctrl: \x89 P N G \r \n \x1A \nP N Gin the middle is obvious. The other five bytes are a booby trap for 1995-era file transfer, and each one catches a specific failure:0x89has the high bit set. Some 7-bit transfer paths stripped bit 7 from every byte. If that happened, this byte arrives as0x09and the file is detectably wrong on byte one.0x0D 0x0Ais a DOS line ending. A text-mode FTP transfer that “helpfully” converts CRLF to LF mangles it.0x1Ais Ctrl-Z, the MS-DOS end-of-file marker. If youTYPEa PNG at a DOS prompt, output stops here instead of spraying binary at your terminal and leaving it in a weird state.0x0Ais a bare LF, catching the opposite conversion: LF silently expanded to CRLF.
Who would have thought that so many bits were used just to account for line endings in different operating systems? I suppose it’s good to plan ahead when designing a file format.
Everything Is a Chunk
After the signature, a PNG is a flat sequence of chunks. No central directory, no offset table. You read them in order.
Every chunk has the same four-field shape:
Field Size Notes Length 4 bytes Big-endian, counts only the data field Chunk Type 4 bytes Four ASCII letters Chunk Data Length bytes Can be zero-length CRC-32 4 bytes Computed over type and data, not over length Two details worth keeping. The length field is 32 bits but the spec caps values at 2³¹−1, so the high bit is always clear. And the CRC covers the type plus the data but skips the length, which means a corrupted length field is not detected by the chunk’s own checksum.
The chunk type is where PNG does something clever. Those four letters are ASCII, and bit 5 of an ASCII letter is what distinguishes uppercase from lowercase (
Ais0x41,ais0x61). PNG uses that bit in each of the four positions as a flag:Position Uppercase means Lowercase means 1st Critical: decoder must understand it Ancillary: safe to ignore 2nd Public, registered in the spec Private, vendor-specific 3rd Reserved, must be uppercase today (reserved for future use) 4th Unsafe to copy if pixels changed Safe to copy blindly So a decoder that has never heard of
tEXtcan tell from the lowercasetthat skipping it is fine. A decoder hittingIDATsees the uppercaseIand knows it cannot skip it. The capability negotiation is encoded in the name itself, which means you can add chunk types decades later without breaking old readers. This is why APNG could bolt animation onto PNG without a version bump.Four chunk types are critical:
IHDR(header, always first),PLTE(palette),IDAT(the pixels), andIEND(a zero-length terminator).
IHDR Is the Decode Key
IHDRis exactly 13 bytes and it comes first because nothing else can be interpreted without it:- Width (4 bytes) and Height (4 bytes), big-endian
- Bit depth (1 byte): bits per sample, one of 1, 2, 4, 8, 16
- Color type (1 byte): what a pixel is made of
- Compression method (1 byte): always 0
- Filter method (1 byte): always 0
- Interlace method (1 byte): 0 for none, 1 for Adam7
Bit depth and color type together determine everything about the pixel layout, and only certain combinations are legal:
Color type Name Samples per pixel Legal bit depths 0 Greyscale 1 1, 2, 4, 8, 16 2 Truecolor (RGB) 3 8, 16 3 Indexed 1 (a palette index) 1, 2, 4, 8 4 Greyscale + alpha 2 8, 16 6 Truecolor + alpha (RGBA) 4 8, 16 Note the gaps. You cannot have 16-bit indexed color, because a palette holds at most 256 entries and 8 bits already addresses all of them. You cannot have 1-bit RGB, because a “1-bit red sample” isn’t a useful thing. The table isn’t arbitrary; each missing cell is a combination that would be incoherent.
Also note that bit depth is per sample, not per pixel. A bit depth of 16 with color type 6 means 16 bits each for R, G, B, and A: 64 bits per pixel. That’s the “64-bit RGBA” you see in PNG marketing.
Where the Pixels Actually Live
Uncompress all the
IDATdata and concatenate it, and you get a byte stream. That stream is not a grid. It’s a sequence of scanlines, one per image row, top to bottom. And each scanline is:[1 filter type byte][packed sample data for the whole row]That leading byte is not pixel data. It’s a number from 0 to 4 saying which filter was applied to this row.
The sample data is packed with no padding between pixels and no separators. Samples appear in a fixed order within each pixel:
- Greyscale:
grey - Truecolor:
red, green, blue - Indexed:
palette index - Greyscale + alpha:
grey, alpha - Truecolor + alpha:
red, green, blue, alpha
Here is an example PNG, filter type 0 (no filtering) on both rows:
scanline 0: 00 ff 00 00 00 ff 00 00 00 ff ff ff 00 ^^ filter byte ^^^^^^^^ red pixel (ff,00,00) ^^^^^^^^ green pixel (00,ff,00) scanline 1: 00 00 00 00 80 80 80 ff ff ff ff 00 ff ^^ filter byte ^^^^^^^^ black ^^^^^^^^ whiteTwelve bytes of pixel data per row (4 pixels × 3 samples), each prefixed by one filter byte, for 26 bytes of raw stream. The complete file, signature and all four chunks included, is 83 bytes.
The whole model at 8-bit depth: walk the row, emit samples in order, move on. No alignment, no padding, no per-pixel headers.
Below 8 Bits, Pixels Share Bytes
Bit depths of 1, 2, and 4 only apply to greyscale and indexed images. Multiple pixels get packed into a single byte.
These samples are packed into bytes with the leftmost sample in the high-order bits of a byte followed by the other samples for the scanline.
Leftmost pixel goes in the high bits. So for a 12-pixel-wide 1-bit greyscale image:
pixels : 1 1 0 1 0 0 0 1 1 0 1 1 packed bytes : 0xd1 0xb0 11010001 10110000 ^^^^ unusedTwelve pixels need 12 bits, which rounds up to 2 bytes, leaving 4 bits spare at the end. The spec’s language on those leftover bits:
When there are multiple pixels per byte, some low-order bits of the last byte of a scanline may go unused. The contents of these unused bits are not specified.
Scanlines always start on a byte boundary. Row 2 never continues in the leftover bits of row 1’s last byte.
At bit depth 16, each sample is two bytes, most significant byte first. The spec calls it network byte order. On x86 and ARM, which are little-endian, that means every 16-bit sample needs a byte swap on read and on write.
The Filter Byte Is the Whole Trick
Now back to that leading byte on every scanline.
PNG uses DEFLATE, the same algorithm as gzip and zip. If you just DEFLATE’d raw pixels, PNG would compress about as well as gzipping a bitmap, which is to say barely at all. Photographs and gradients don’t repeat exact byte sequences, and LZ77 needs exact repeats.
So before compressing, PNG transforms each scanline into differences from its neighbors. Five filters are available, chosen per scanline:
Type Name Transform 0 None store the byte as-is 1 Sub subtract the byte from the pixel to the left 2 Up subtract the byte from the pixel above 3 Average subtract the average of left and above 4 Paeth subtract whichever of left/above/upper-left is the best predictor All arithmetic is mod 256, which is what makes it reversible without storing a sign. And “the pixel to the left” means the byte at the same position in the previous pixel, so for RGB the red sample is compared against the previous red sample, not against the previous blue.
Take a 16-pixel greyscale gradient stepping by 10:
raw scanline : 00 0a 14 1e 28 32 3c 46 50 5a 64 6e 78 82 8c 96 after Sub filter : 00 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0aSixteen distinct byte values become two. The image is unchanged and the transform is exactly reversible, but LZ77 now sees a run it can encode in almost nothing. On this toy row, DEFLATE produces 28 bytes for the raw version and 15 for the filtered one.
Sixteen bytes is far too small for DEFLATE to stretch its legs, so don’t read that ratio as typical. The point is the entropy collapse: filtering doesn’t compress anything, it rearranges the data so the compressor has something to find.
That per-scanline choice is also why two encoders produce different-sized files from identical pixels. libpng, ImageMagick,
oxipng, andzopflipngall ship different filter-selection heuristics. Same spec, same decoded output, different bytes on disk. Most PNG optimizers are search algorithms over filter choices, not better compressors.
The Compression Pipeline, End to End
Putting it together:
raw pixels -> pack into scanlines (samples in order, sub-byte packing if needed) -> prepend a filter byte per scanline, apply the filter -> DEFLATE the whole concatenated stream (LZ77 + Huffman) -> wrap in a zlib container (RFC 1950) -> split across one or more IDAT chunksA few consequences fall out of that ordering:
The zlib stream spans chunks.
IDATboundaries are arbitrary. A decoder must concatenate everyIDATpayload and then decompress; decompressing them individually fails. Encoders split them for streaming, not for structure.The Adler-32 checksum in the zlib wrapper covers filtered bytes, not your original pixels. It validates decompression, not image fidelity. The per-chunk CRC-32 is what protects against transmission corruption.
Compression is global across the image. LZ77’s 32KB sliding window means row 400 can match against row 380 if they’re similar. This is why a 64×64 solid color block compresses to 136 bytes while a 64×64 gradient of the same dimensions takes 10,362 bytes, against 12,288 bytes raw. Uniformity compresses; novelty doesn’t.
And a practical one: for that solid-color block, encoding as indexed color with a one-entry palette produces a 99-byte file instead of 136, because each pixel is one index byte instead of three samples. If your image has few colors, color type 3 usually beats truecolor even after DEFLATE gets its turn.
Interlacing, Briefly
If the interlace byte in
IHDRis 1, the image uses Adam7: the pixels are transmitted in seven passes over an 8×8 grid, coarse to fine, so a partially-downloaded image renders as a low-resolution preview that sharpens.Two things to know. Each pass is filtered and encoded as an independent sub-image with its own scanlines and filter bytes, so a decoder can’t treat the stream as one grid. And Adam7 typically makes files larger, because breaking the image into seven sparse sub-images destroys exactly the local coherence that filtering and LZ77 depend on. It was a good trade on a 28.8k modem. On any modern connection it costs size and complexity for a progressive render nobody waits around to see.
What This Buys You
The design decisions hold up well for a 1996 format:
- Unknown chunks are safe by construction, so the format extended to EXIF metadata, ICC profiles, and animation without ever breaking old decoders.
- Every chunk is individually checksummed, so corruption is localized and detectable rather than silently rendering garbage.
- Filtering is a preprocessing step, not a compression format, which means encoders can get better forever without touching the spec. A file written by
zopflipngtoday decodes fine in a 1997 reader.
That extensibility is not just historical. PNG got a Third Edition as a W3C Recommendation on 24 June 2025, which finally standardized APNG, added an
eXIfchunk for camera metadata, and brought in HDR through three new chunks (cICP,mDCV,cLLI). Thirty years on, the container still had room.Where it shows its age is DEFLATE, which is a 1990s compressor. Lossless WebP does beat it: Google’s own study puts WebP lossless at 23% smaller than PNGs already optimized with ZopfliPNG, and 42% smaller than default libpng output. Worth noting the baseline matters enormously there, and Google’s WebP FAQ quotes a different figure (26%) than the study it links to.
Lossless AVIF is a murkier story than the marketing suggests. AOMedia publishes no general lossless-AVIF-versus-PNG number at all; its quantified claims (50% versus JPEG, 30% versus WebP) are all about lossy encoding. The only primary figure available is 10% versus a 16-bit PNG for a single demo image using a new v1.2.0 feature. Independent testing regularly finds lossless AVIF producing larger files than PNG for flat synthetic images like icons, UI, and charts. If you’re picking a format for screenshots and diagrams, test on your own images rather than trusting a general ranking.
Next in the series: the opposite of all this. A text file, which announces nothing about itself at all.
Sources
- W3C PNG Specification, Third Edition — the current standard; §7.2 covers scanlines and sample packing, §9 covers filtering
- RFC 2083 — the original 1997 IETF PNG specification
- RFC 1950 (zlib) and RFC 1951 (DEFLATE) — the compression layer
- libpng PNG history — the Unisys announcement, the PBF name, and the January 1995 timeline
- WebP Lossless and Alpha Study — Google’s 23%/42% lossless figures and their baselines
- AOMedia on AVIF v1.2.0 — the 10% lossless figure, and its narrow scope
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Where Should This Agent Knowledge Live?
Every agent has a junk drawer.
It usually starts with project instructions. Then I added build commands, personal preferences, database warnings, old architecture decisions, and things to fix later.
The agent could see everything if I wanted it to, but then it would have to read a small novel before touching the code, recurring workflows were buried between random facts, and completed work kept hanging around like it was still relevant. I had given the agent more context and somehow made it less informed.
The problem was not missing knowledge. The problem was putting every kind of knowledge in the same place. An instruction, a skill, a memory, and an issue can all be written in Markdown.
They do four completely different jobs.
Four Places, Four Jobs
A clean version looks like this:
What the agent needs Where it belongs A rule that must apply during every relevant session Instructions A reusable procedure for a particular kind of work Skill A durable fact that may become relevant later Memory A commitment that remains open until completed Issue tracker In practice, this is messier than a table… That’s where the engineering and attention to detail really matter.
So the useful questions are Does the agent need to know about this? and What bucket does this knowledge belong in?
Instructions Are Guaranteed and Expensive
Project instruction files are the things your agent loads automatically. Depending on the tool, that might be
AGENTS.md,CLAUDE.md, or another repository-level file.This is your guaranteed layer. The agent (or harness) doesn’t have to remember to search for it. If a session starts in the project, the rules are sitting in context.
Use that guarantee for knowledge that must shape nearly every relevant action:
- the preferred package manager and command runner,
- where the main source and tests live,
- dangerous commands that require explicit approval,
- the authoritative source for important data,
- mandatory validation before work counts as complete,
- a pointer telling the agent when to load a skill or recall a memory.
The guarantees come with a cost. Every line added to the context is loaded on every session, even when all you really need is a lightweight session where that context doesn’t matter.
Be diligent about cleaning up and maintaining your guaranteed context window, especially if you don’t have a memory layer in place.
Skills Are Procedures With Judgment
A skill answers a different question: how should the agent perform this kind of work?
Publishing a blog post, reviewing a pull request, applying a database migration, preparing a release, updating dependencies. Those are workflows. They have an entry condition, a sequence, safety rules, and a way to verify the result. That’s more than a fact. It’s operational judgment packaged for reuse.
Before we had skills, we had playbooks. Now we can make playbooks out of anything.
A good skill tells the agent when the workflow applies, what to inspect before acting, which steps and tools are appropriate, what must never happen silently, and what evidence proves the work succeeded.
Maybe the deployment instructions can now stay short; when doing a deployment, load the deployment skill.
Instructions are guaranteed. Skills are conditional.
Memory Is Context, Not Policy
Memory is where durable facts live without being injected into every session.
I prefer pnpm for JavaScript and most TypeScript projects. I prefer uv, and sometimes Poetry, for Python. These are facts that shouldn’t have to be repeated.
What about that time you had to troubleshoot an integration and observed some strange behavior? What about when you changed the database design and it broke the support layer? None of this deserves to be injected into every prompt, but it deserves a place where the details can be accessed later.
A semantic memory system can store and retrieve the relevant durable facts and give them to the agent when it asks. I described this earlier.
Memory can be large, and it can be flexible. But it’s also not guaranteed. The agent might not use the right keyword. You might have a problem with the vendor. A critical dependency could go down and take the memory system offline.
Don’t put safety-critical policies in memory. It’s good to have backups. If preferences get lost, they can be recreated, but absolutes like never print secret values belong in several places. If it has anything to do with security, cover your ass.
Memory is best for facts, preferences, relationships, explanations, and past decisions.
Issues Are Promises, Not Storage
An issue tracker tells the agent what needs to be done.
Issues have always been little documentation vaults. We write the history of the bug as it travels through the system. We link back to the issue as it maintains relevance.
An issue should preserve the context for a decision. It should act as a durable property of the project, recording the circumstances around a decision point.
Don’t make it a container for everything the agent did along the way, but I think it’s totally fine if you use it to publish an implementation plan.
Just, you know, you gotta read it.
Our job now is reading about software. The issue trackers are our corpus.
Route the Knowledge With Four Questions
When I don’t know where something belongs, these four questions can help.
1. Must the agent know this before it acts?
Instruction
2. Is this about performing a recurring kind of work?
Skill
3. Is this a durable fact that may help later?
Memory
4. Is this unfinished work or a commitment?
Issue
Sometimes the answer can be more than one place. But don’t copy the content blindly between locations.
All the files may be Markdown, but maintaining the architecture now means knowing where to put the information.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Every Feature Opens a Maintenance Account
Coding agents have developed a dangerous little habit. You ask for one change, and somewhere near the end they offer to add three more.
Would you like a configuration flag? A JSON output mode? A retry option? Maybe a dashboard while we’re here?
The agent can build it. The tests and relevant files are already in the context. So why not?
Just one more feature, one more prompt. You know what I’m talking about.
Then six months later the flag is still there. Somebody relies on the JSON shape. The retry behavior conflicts with a new timeout. The dashboard has a dependency that needs a security update. The agent that offered to build all of it has moved on with its life, mostly because it never had one.
You still own the feature.
That’s the part we need to consider. Every feature is another record in a maintenance ledger.
A Feature Is a Standing Promise
We tend to think of a feature as the code that implements it. Add the function, connect the interface, write the tests, merge the change. Feature complete.
That’s the construction phase. The feature itself is a promise that begins after the merge.
If you add a CLI flag, you’re promising that scripts can keep using it. If you expose a JSON response, you’re promising something about its shape. If you store a new piece of data, you’re promising to preserve, migrate, export, and eventually delete it correctly. If you connect another service, you’re promising to notice when its API changes.
The code might be thirty lines. The promise has no line count.
This is why tiny features get expensive. The implementation fits in one file, but the behavior crosses documentation, tests, support, deployment, security, and every future change near that boundary. Agents are very good at showing us the thirty lines. They’re much less likely to show us the next three years.
The Opening Balance Looks Great
Before coding agents, implementation cost acted as a filter. Not always a good one, but a filter.
Someone had to estimate the work, find time in a sprint, write the code, and get it reviewed. A small convenience feature might lose simply because nobody wanted to spend two days on it. Frustrating, sure, but it forced the question: is this worth building?
Now the estimate is twenty minutes. The agent has already inspected the codebase. It can update the model, add the command, generate the tests, and fix the type errors without needing another meeting. The old cost-benefit calculation collapses, because almost everything looks worth building when you only count the first implementation.
So we say yes more often.
That’s not automatically bad. Plenty of useful software never existed because construction cost too much. Cheaper implementation lets small teams solve problems that used to require a real budget. Good for us, but the maintenance math didn’t collapse along with it.
The feature still adds another path through the system. It still creates behavior that can regress. It still has users, even if the only user is you on a Sunday afternoon six months from now.
The opening balance is cheap. The account stays open.
What Accumulates
Maintenance is easy to wave away because no single piece sounds overwhelming. It’s just one more test. One more paragraph in the docs. One more migration. Then the interest starts adding up:
- Compatibility: Existing callers depend on behavior you considered an implementation detail.
- Testing: Every supported path needs coverage, fixtures, and updates when neighboring code changes.
- Documentation: The feature needs to be discoverable, accurate, and removed from the docs if it goes away.
- Dependencies: A tiny feature can introduce a library that now participates in every upgrade and security review.
- Operations: New jobs, tables, queues, or API calls need logs, failure handling, and a recovery story.
- Support: Someone has to answer why it behaved differently on another machine.
- Removal: Deleting it later means finding its users, migrating their data, and deciding how long compatibility lasts.
None of these costs are unique to generated code. We’ve always paid them. The difference is volume. Agents let us open maintenance accounts much faster than we close them.
A Ten-Minute Flag Is Still an Interface
Let me give you an example.
You have a command that prints a human-readable table. An agent offers to add
--json. That sounds great. It probably is great. The code serializes the existing records, the tests compare a sample payload, and the whole change lands before lunch.Then someone pipes that output into another script.
Now field names matter. Null behavior matters. Ordering might matter even though you never promised it. A renamed internal property breaks an external workflow. Adding a timestamp creates noisy diffs. Removing a field requires a compatibility decision.
The flag didn’t add another display format. It created an API.
Would you still build it? Probably. I like useful CLI tools, and machine-readable output is usually worth supporting. The point isn’t to reject the feature. The point is to recognize the account you’re opening. Once you see it as an interface instead of a ten-minute patch, you define the schema deliberately, document what’s stable, avoid exposing fields that should stay internal, and decide whether versioning matters before somebody’s automation answers that question for you.
Same code. Better ownership.
Backlogs Hide the Statements
One reason maintenance gets away from us is that backlogs are organized around changes, not promises.
The issue says “add export support.” It rarely says:
Maintain this export format for as long as anyone depends on it, update it whenever the underlying model changes, keep its documentation accurate, and provide a safe way to retire it later.
That would look ridiculous in an issue title. It’s still what the issue means.
Agents make backlogs disappear quickly, which feels fantastic. I’ve watched them knock out work that would have sat around for months. But a closed issue can become an open obligation. A project with fifty completed features isn’t necessarily healthier than one with twenty. It might just have thirty more things that can break.
Price the Account Before You Open It
I don’t want a meeting for every CLI flag. The whole advantage of these tools is that we can move fast.
We can still take thirty seconds to ask better questions before accepting the extra code:
- Who will depend on this? A person clicking a button creates a different promise than a script parsing output.
- What new state or interface does it introduce? Stored data and public schemas are much harder to remove than local calculations.
- What has to stay compatible? Name the stable boundary instead of letting users guess.
- How will we know it broke? Tests help, but logs, validation, and recovery may matter more.
- What ongoing work does it create? Dependencies, docs, migrations, provider changes.
- What would cause us to close the account? Decide now whether it’s experimental, permanent, or removable.
If the answers are cheap too, build it. If the feature creates a permanent public contract for a minor convenience, nope, not going in.
Cheap Construction Needs Better Restraint
I’m not interested in making software expensive again. Faster implementation is good. More people turning an idea into a working tool is good. Small teams getting leverage that used to belong to large companies is very good.
We just need to stop treating features as free.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
AI Programming Software-development Coding-agents Maintenance
-
Pi and Hermes Are Trying to Solve Different Problems
I went looking for a talk from Mario Zechner, the creator of Pi, because I wanted to understand why someone would build another coding agent when we already have a pile of them. I found: a talk called “Building pi in a World of Slop.”
Zechner described Pi as a minimal, extensible coding agent that should fit your workflows instead of forcing you into its workflow. He also made a point that should be printed on the box of every AI coding tool: code is not free. The model can produce it quickly, sure. You still own the review, the maintenance, the weird edge cases, and the next person trying to understand it six months later.
That framing explains Pi better than any feature list does.
I’ve also been reading about Hermes Agent, from Nous Research. Hermes is a useful comparison because it’s also an open, provider-flexible agent harness. But it isn’t trying to be Pi with a few extra switches turned on.
Pi and Hermes are trying to solve different problems.
Pi Gives You a Small Place to Start
Pi is a terminal coding harness with a deliberately small default: read files, write files, edit files, run shell commands. Underneath that CLI is a set of TypeScript packages for model access, the agent loop, sessions, and the terminal UI. You can use the CLI, run it through JSON/RPC, or embed the SDK in something else.
That last part is the point.
Pi deliberately leaves out things a lot of agent products treat as table stakes: MCP in the core, subagents, plan mode, permission popups, to-do lists, background shell work. This can look like a missing-feature list if you evaluate it like Claude Code or another finished product.
I don’t think that’s the right test.
Those omissions are Pi’s design. It’s saying: a harness should give you a stable loop, a tool boundary, sessions, and enough extension points to build the workflow you actually need. Then it should get out of the way.
Want MCP? Add it. Want a planning workflow? Make one. Want agents that coordinate over a message bus, work in separate git worktrees, or run in a weird internal deployment? You own the composition. Pi has extensions and packages for that, and now an explicitly experimental orchestration package, but none of it is presented as the one true way to work.
That’s a compelling idea if you’re building a specialized system. It’s also work. Both things can be true.
Hermes Starts With the System
Hermes starts from almost the opposite direction. It’s an integrated autonomous-agent platform with persistent memory, learned skills, built-in delegation, MCP support, scheduling, multiple execution environments, and surfaces that extend beyond the terminal into messaging and desktop interfaces.
Hermes is asking a larger question: what does an agent need to keep working over time, across channels, with memory of what it has already learned?
That’s not just a bigger Pi configuration.
When Hermes includes persistent memory and skill creation, it’s making those things part of the product contract. When it includes subagents and scheduling, it’s giving you an operating model for delegation and recurring work. You get more out of the box, and you inherit more of the system’s assumptions.
For a lot of people, that’s exactly right. If you want an agent to run continuously, show up in Slack or Telegram, remember prior work, and execute recurring workflows, building all of that from Pi primitives would be a very committed hobby.
Good for you, but I think most teams shouldn’t volunteer for that job unless the control model is part of what they’re building.
The Comparison That Matters
Here’s the version I keep coming back to:
Pi Hermes Default posture Minimal programmable harness Integrated autonomous-agent platform Core workflow You compose the pieces The product ships an opinionated system Multi-agent work Extensions, packages, or your own topology Built-in delegation and parallel work Memory Session primitives and JSONL history Persistent memory and skill-learning features Best fit A workflow or control plane you need to own A capable agent system you want to operate This isn’t a scorecard. Hermes isn’t “better” because it has more rows filled in, and Pi isn’t “purer” because it has fewer.
The question is where you want the complexity to live.
With Pi, much of it lives in the system you build around the harness. You have to decide how agents coordinate, what gets remembered, which tools are safe, and how approval works. In exchange, the result can fit your environment instead of being a very configurable version of someone else’s environment.
With Hermes, more of that complexity is already in the platform. You spend less time assembling basic capabilities, but you should understand its memory model, delegation model, security posture, and operational boundaries before you give it real work.
Neither choice removes responsibility. It just changes the shape of it.
Don’t Build a Harness Because It Sounds Fun
Agent harnesses are one of those things that sound like a great weekend project. You wire up a model, give it a few tools, add memory, spawn a couple subagents, and suddenly you have a tiny digital organization running in your terminal.
Then Monday happens.
The agent needs a permission model. It needs observability. It needs a way to recover from bad state. It needs sensible defaults for credentials and logs. It needs evaluation. It needs someone to own the changes when a provider API shifts or an extension becomes a security problem.
That’s why I like the Pi and Hermes comparison. It makes the tradeoff visible.
Use Hermes when you want an agent platform. It already has an opinion about the features an always-on, multi-surface agent needs.
Use Pi when the workflow itself is the product, or when the product assumptions are exactly what you need to escape. Pi’s small core is valuable because it leaves room for a different control plane.
And if all you need is a better code-review prompt or a way to query one internal system, build that inside the harness you already use. A skill, extension, or MCP server is usually a better answer than inventing an agent platform because you wanted one new capability.
This is the same point I landed on in a recent post: you think you want to build your own harness, but what you usually want is a wrapper around the one you already have.
Code is not free. Neither is a harness.
Sources & References
- “Building pi in a World of Slop” — Mario Zechner (talk) — Pi’s design philosophy, workflow fit, and the cost of generated code.
- Pi documentation — current product scope, installation, extensions, and operating modes.
- Pi usage documentation — default tool surface and deliberate core omissions.
- Pi monorepo — TypeScript package architecture and experimental orchestrator package.
- Hermes Agent documentation — persistent memory, skills, delegation, MCP, execution environments, and surfaces.
- Hermes Agent repository — open-source project and implementation reference.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Graphify Turns Your Repos Into a Map You Can Query
Navigating code dependencies inside a single repository is already hard enough. But if you’re on a microservice setup, or a split frontend and backend, tracking what depends on what across multiple repos is a special kind of misery. A backend API route changes. Which frontend components just broke? Good luck. You’re grepping three workspaces and hoping you didn’t miss one.
So when I ran across Graphify, an open-source project from Graphify Labs (YC S26), it caught my attention. It maps your code directories into queryable knowledge graphs. Not fuzzy text search. Not an expensive vector RAG lookup that burns tokens every time you ask it a question. A deterministic index of your codebase.
Let me walk through how it works, why it’s useful for AI coding agents, and the part I wanted to figure out: how to stitch multiple repos into one unified map.
What Graphify Does
Instead of guessing at relationships, Graphify parses your source and builds a real graph out of it. Three pieces make it tick:
- Deterministic AST parsing. It uses
tree-sittergrammars locally to parse roughly 40 languages, pulling out classes, functions, calls, and imports. No LLM tokens, no API rate limits. Just parsing. - Explicit vs. inferred edges. Every relationship gets a confidence tag.
EXTRACTEDmeans it’s right there in the syntax, like an import or a direct function call.INFERREDmeans it deduced the connection from context. You always know how much to trust an edge. - Leiden community clustering. It automatically segments your code into logical domain boundaries, which makes it easy to spot the “god nodes”, the files with way too many dependencies hanging off them. Those are usually the first thing you want to refactor.
Merging Multiple Repos Into One Graph
This is the part I cared about. Graphify supports it natively through the CLI, and here’s the flow straight from the docs (I haven’t run it on my own repos yet). Say you’ve got a frontend repo and a backend repo. Three steps.
Step 1: Scan each repo on its own. Run the scan inside each folder. Results land in a
graphify-out/directory.# In your frontend repo cd ~/Work/frontend graphify . # In your backend repo cd ~/Work/backend graphify .Step 2: Merge the graphs. The
merge-graphssubcommand joins the JSON outputs into one combined map of nodes and relationships.graphify merge-graphs \ ~/Work/frontend/graphify-out/graph.json \ ~/Work/backend/graphify-out/graph.json \ --out ~/Work/combined_graph.jsonStep 3: Traverse it, or hand it to your agent. Now you can trace a call path straight across the service boundary, or serve the combined graph to a coding agent over MCP.
# Trace a path across the frontend/backend boundary graphify path "login_component.ts" "auth_controller.py" --graph ~/Work/combined_graph.json # Or expose the combined graph to your coding agent over MCP python -m graphify.serve --graph ~/Work/combined_graph.jsonThat
pathcommand is the whole pitch, honestly. You point it at a frontend file and a backend file and it tells you how they’re connected. No manual grep archaeology.Why This Matters for AI Coding Agents
If you use Claude Code, Cursor, or Antigravity, you already know the problem. Feed the agent raw files and you torch the context window in about four prompts. Point it at Graphify’s output instead, the
GRAPH_REPORT.mdor thegraph.jsonover MCP, and the agent can do a few things it otherwise can’t:- Figure out exactly which files a refactor will touch before it edits anything.
- Trace dependency lineage across code boundaries deterministically, not by vibes.
- Describe your architecture based on the actual shape of the code, not a hallucinated version of it.
That last one is underrated. Half of “the AI got confused” moments happen because the AI never saw the whole picture.
Two Gotchas Before You Install
A couple of things will trip you up, so here they are up front.
The package name has a typo built in.
graphifywas already taken on PyPI, so the official package is registered asgraphifyy. Two y’s. You install it like this:pip install graphifyyWatch your Python version. The Leiden community detection library has C-extension limits, so Graphify currently runs best on Python under 3.13. Worth checking or switching to a compatible version (like 3.12) using mise.
The honest appeal here isn’t the visualization, pretty as the HTML map is. It’s that cross-repo dependency tracing has been a manual, error-prone chore for as long as I’ve worked on split codebases, and this makes it a single command.
Sources
- Graphify Labs on GitHub: setup requirements, supported parsers, and CLI options.
- Auriga IT’s Graphify introduction: explains the three-pass architecture and Leiden clustering optimization.
- Graphify on PyPI: package installation details and version compatibility.
- Aider’s Repository Map: on using tree-sitter to parse AST-based codebase maps for token-efficient coding context.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Deterministic AST parsing. It uses