LLBBL Blog

JPEG Quality 80 Is Not a Setting

Mon, 24 Aug 2026 10:00:00 -0500

MP4’s boxes are exact about everything: every byte accounted for, every offset written down. JPEG is exact about its structure too, and then hands you one control that has no defined meaning at all.

It destroys things. That is the entire point, and it will not tell you how much.

Here is one 512x512 PNG, encoded to JPEG four times, on the same machine, with every encoder set to quality 80.

q80-cjpeg.jpg      42595 bytes    (libjpeg-turbo 3.2.0)
q80-pillow.jpg     42595 bytes    (Pillow 12.3.0)
q80-sips.jpg       67865 bytes    (macOS sips)
q80-ffmpeg.jpg     13847 bytes    (ffmpeg -q:v 80)

Same input. Same number typed into the same-named parameter. A 5x spread in output size.

The Number Is an Index, Not a Measurement

There is no quality field in a JPEG file. Nothing in ISO/IEC 10918-1 defines a scale from 0 to 100. What the file actually carries is a quantization table: 64 integers that every DCT coefficient in an 8x8 block gets divided by before rounding. Bigger divisors, more coefficients rounded to zero, smaller file, more damage.

“Quality 80” is just a name your encoder gives to one particular table. Here is the first row of the luminance table each of those four files chose:

cjpeg    6   4   4   6  10  16  20  24
pillow   6   4   4   6  10  16  20  24
sips     2   2   2   3   4   5   7   8
ffmpeg   8  62  73  85 100 104 112 131

cjpeg and Pillow are identical because Pillow links libjpeg and inherits its table. Both are the standard Annex K example table scaled by a linear formula: at quality 80 the scale factor is 40%, and 16 x 0.40 rounds to 6, 11 x 0.40 rounds to 4, and so on down the row.

macOS sips divides by 2 where libjpeg divides by 6. Its quality 80 lands somewhere around libjpeg’s 93, and it is not the standard table scaled differently, it is a different table. Apple picked their own numbers.

ffmpeg is the funny one. Its -q:v for MJPEG runs 2 to 31, where lower is better. So -q:v 80 gets clamped to 31, the worst setting it has:

$ ffmpeg -i source.png -q:v 31 f31.jpg
$ cmp f31.jpg q80-ffmpeg.jpg
$

Byte-identical. Asking ffmpeg for 80 asks it for the ugliest image it knows how to make, and it does not warn you, because 80 is a perfectly valid thing to say to a parameter that happens to top out at 31.

None of these encoders is wrong. The specification never told them what 80 means.

The Damage Is Mostly Not Where You Think

The DCT-and-quantize step gets all the attention. It is not usually the thing wrecking your image.

Before any of that happens, the encoder converts RGB to YCbCr and then, by default, throws away three quarters of the color information. 4:2:0 subsampling averages the two chroma channels over 2x2 pixel blocks. Human vision is much less sensitive to color detail than to brightness detail, so most of the time you cannot see it.

Most of the time. Here is a 400x120 image, pure red on pure blue, encoded at quality 95 both ways:

                    size      max channel error   mean error
4:4:4 (no subsampling)   19688 bytes          20         0.59
4:2:0 (default)          10782 bytes         232        14.70

Quality 95 is a setting people reach for when they want the image to be basically untouched. At 4:2:0 a single channel is off by 232 out of 255. The red and blue have the same luminance, so the entire edge between them lives in chroma, and chroma is the part that got averaged away.

This is why red text on a colored background looks like it was scanned by a fax machine, why UI screenshots with colored syntax highlighting come out muddy, and why a logo saved as JPEG at “high quality” still has a smeared halo. Turning subsampling off costs about 80% more bytes here and takes the error from 232 to 20.

cjpeg -sample 1x1 does it. In Pillow it is subsampling=0. Almost no tool exposes it in a GUI, and almost every default is 4:2:0.

Generation Loss Converges

The folk wisdom is that re-saving a JPEG degrades it a little more each time, forever, until it turns to soup. I re-encoded the same image 50 times at quality 85, decoding and re-encoding each round, and measured the drift from the original:

gen   size(bytes)   mean err   max err
  1        48911       3.51        194
  2        49532       4.11        188
  5        49351       5.21        197
 10        49299       6.17        199
 25        49249       7.20        190
 50        49189       7.54        190

Most of the loss happens on save one. Generation two adds about half a point. Generations 25 through 50 add a third of a point between them, and the maximum error never moves at all.

It converges because quantization is idempotent once you land on the grid. Decode a coefficient that was rounded to 6 times its divisor, transform it back, and it quantizes to the same bucket. The image reaches a fixed point that survives re-encoding. What breaks this is changing anything: a different quality, a different subsampling mode, a crop that shifts the 8x8 block boundaries, or a rotation that resamples. Then you land on a new grid and pay the first-generation cost again.

Which is the actual reason jpegtran exists:

$ jpegtran -rotate 180 -outfile r1.jpg original.jpg
$ jpegtran -rotate 180 -outfile r2.jpg r1.jpg
$ cmp original.jpg r2.jpg
$

Two 180-degree rotations, byte-identical to the input. jpegtran permutes the already-quantized coefficient blocks without ever decoding to pixels, so there is nothing to re-quantize. Rotating, flipping, and cropping on 8-pixel boundaries are all lossless if you use the right tool. Almost nobody does.

What the File Looks Like

Worth thirty seconds, because the structure is unusually clean. Every marker is 0xFF followed by a type byte, and every marker except the two bare ones carries a 2-byte big-endian length:

$ xxd -g 1 -l 32 q80-cjpeg.jpg
00000000: ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01  ......JFIF......
00000010: 00 01 00 00 ff db 00 43 00 06 04 05 06 05 04 06  .......C........

ffd8 is Start of Image. ffe0 is APP0, length 0x0010, containing the literal string JFIF\0. Then at offset 20, ffdb is Define Quantization Table, length 67, table 0, and the bytes after it are the 64 divisors in zigzag order. 06 04 05 06 05 04 is that first row I printed above, read diagonally.

Walking the whole file:

     0  FFD8  SOI
     2  FFE0  APP0  length 16
    20  FFDB  DQT   length 67
    89  FFDB  DQT   length 67
   158  FFC0  SOF0  length 17
   177  FFC4  DHT   length 31
   210  FFC4  DHT   length 181
   393  FFC4  DHT   length 31
   426  FFC4  DHT   length 181
   609  FFDA  SOS   length 12
 42593  FFD9  EOI

Everything structural fits in the first 623 bytes. The remaining 42,000 are entropy-coded scan data with no framing at all, which creates one last problem: if a Huffman code happens to emit the byte 0xFF, a decoder scanning for markers would misread it. So the encoder stuffs a 0x00 after every literal 0xFF, and the decoder throws it away. This file contains 504 of those.

What To Do About It

Never move a quality number between tools. Quality 80 in Photoshop, cjpeg, sips, and ffmpeg are four unrelated things. If you are porting a pipeline, re-tune by measuring output size or error, not by copying the integer.
Turn off chroma subsampling for anything with saturated color edges. Logos, screenshots, charts, text. -sample 1x1 in cjpeg, subsampling=0 in Pillow. Leave 4:2:0 on for photographs, where it is nearly free.
Use jpegtran for rotations and crops. -rotate, -flip, -crop, and -perfect operate on coefficients and cost nothing.
Stop worrying about generation loss and start worrying about generation one. The first save is where the damage is. Keep the original.
Don’t put a JPEG in the middle of a pipeline. Every intermediate step should be PNG or the raw source. JPEG is an output format.
Strip the metadata deliberately. APP1 holds Exif, which holds GPS coordinates, camera serial numbers, and an embedded thumbnail. Some editors update the pixels and leave the old thumbnail in place, so the crop you made survives only in the big version.

The interesting thing about JPEG is not that it is lossy. Everyone signed up for lossy. It is that the one control the format exposes to users, the quality number, is not part of the format, has no defined meaning, and is quietly reinterpreted by every tool that offers it. You are not setting the quality. You are picking a preset out of a list you cannot see.

Sources

ITU-T T.81 / ISO/IEC 10918-1 — the core JPEG specification; Annex K holds the example quantization tables everybody scales
ITU-T T.871 — JFIF, standardized in 2011, nineteen years after the industry started shipping it
W3C copy of the JFIF 1.02 specification — the original C-Cube document from September 1992
Independent JPEG Group — libjpeg, whose quality-to-table formula became the de facto meaning of the number
libjpeg-turbo — what almost everything actually links against today

An AVIF Is an MP4 With One Frame

Sun, 23 Aug 2026 10:00:00 -0500

GIF’s structure was a header, some tables, and a sequence of blocks with a one-byte type code. MP4 is stricter than that, and the strictness is the whole design. Every byte in the file lives inside a box, and every box starts the same way:

4 bytes   size, big-endian, including this header
4 bytes   type, four ASCII characters

That is it. Eight bytes, and any parser that knows nothing else about the format can walk the entire file, skipping what it doesn’t understand. Some boxes contain other boxes. Some contain payload. There is no data outside a box anywhere in the file.

That design is why the container outgrew video entirely.

Same Header, Different Extension

Here is a ten-second 640x360 H.264 clip:

ftyp  32 bytes  @ 0
free  8 bytes  @ 32
mdat  63602 bytes  @ 40
moov  4424 bytes  @ 63642

Here is a three-second AAC audio file:

ftyp  28 bytes  @ 0
free  8 bytes  @ 28
mdat  26304 bytes  @ 36
moov  1287 bytes  @ 26340

And here is a still image:

ftyp  32 bytes  @ 0
meta  235 bytes  @ 32
  hdlr  33 bytes  @ 44
  pitm  14 bytes  @ 77
  iloc  30 bytes  @ 91
  iinf  40 bytes  @ 121
  iprp  106 bytes  @ 161
    ipco  75 bytes  @ 169
    ipma  23 bytes  @ 244
mdat  17667 bytes  @ 267

That last one is an AVIF. A photograph. It has the same box header format, the same ftyp first, the same mdat holding the payload. What changed is that a still image has no timeline, so instead of moov with its sample tables it uses meta with an item structure: pitm names the primary item, iloc says where in mdat that item’s bytes live, iprp carries its properties.

The ftyp box says which dialect you are reading:

$ xxd -g 1 -l 32 shot.avif
00000000: 00 00 00 20 66 74 79 70 61 76 69 66 00 00 00 00  ... ftypavif....
00000010: 61 76 69 66 6d 69 66 31 6d 69 61 66 4d 41 31 42  avifmif1miafMA1B

Size 32, type ftyp, major brand avif, then four compatible brands: avif, mif1, miaf, MA1B. The .m4a file above declares M4A with isom as a compatible brand. The MP4 declares isom with iso2.

So .mp4, .mov, .m4a, .m4v, .heic, and .avif are one format with six extensions. Your iPhone photo library and your video library are the same container. That happened because in February 1998, ISO picked Apple’s QuickTime file format as the basis for MPEG-4’s container, and the box model turned out to be general enough that everyone who needed a container afterward just used it.

The Box in the Wrong Place

Now the flaw that shaped an entire decade of web video.

The moov box holds the sample tables: which frame starts at which byte, how long it lasts, which chunk it belongs to. mdat holds the frames. A player can decode nothing until it has read moov, because mdat has no internal framing at all. It is one undifferentiated run of bytes, and the only thing that says where frame 0 begins is a number inside moov.

Look at where moov ended up in that first file. Byte 63,642 of a 68,066-byte file.

An encoder writing sequentially cannot know the byte offset of the last chunk until it has written the last chunk, so the natural thing is to write all of mdat and then append moov at the end. That is what nearly every encoder did by default, and it means the player must reach the last 6% of the file before it can show you the first frame.

The fix is a post-processing pass:

$ ffmpeg -i input -c:v libx264 -movflags +faststart out.mp4

ftyp  32 bytes  @ 0
moov  4424 bytes  @ 32
free  8 bytes  @ 4456
mdat  63602 bytes  @ 4464

Same boxes. Same sizes. Same total file length, 68,066 bytes both times, and I checked the mdat payloads byte for byte: identical. All that changed is the order.

It is not quite a memmove, though, because moov’s offsets are absolute positions in the file:

plain.mp4    stco has 1 chunk offsets; first five: [48]
fast.mp4     stco has 1 chunk offsets; first five: [4472]

Moving moov in front of mdat pushed every byte of media 4,424 places later, so every entry in the chunk offset table had to be rewritten by exactly that amount. On a real file with thousands of chunks, that is thousands of pointers, all of which must be corrected, and if the rewrite changes the size of moov (32-bit offsets overflowing into co64) the whole thing has to be recomputed again.

Absolute offsets are the design decision underneath most of MP4’s awkwardness. You cannot concatenate two MP4s. You cannot insert a second of video in the middle. You cannot append to a file that is still being written and have it remain playable. Everything is pointer arithmetic against byte zero.

Fragments Are the Actual Answer

Fragmented MP4 fixes it by giving up on the single index. Instead of one moov describing the whole timeline, you get an initialization segment and then a run of self-describing chunks:

[ ftyp + moov ]  [ moof + mdat ]  [ moof + mdat ]  [ moof + mdat ] ...

Each moof carries the sample table for the mdat that follows it, with offsets relative to the fragment rather than the file. Which means you can start writing before you know how long the video is, cut the stream anywhere, serve any fragment independently, and switch bitrates between fragments without the player noticing.

That property is the entire basis of HLS and DASH. Every adaptive-bitrate stream you have watched is this: a manifest, an init segment, and a pile of moof/mdat pairs that a player stitches together while quietly swapping quality levels based on your bandwidth.

It also means the moov placement problem is now mostly historical for streaming and still completely current for files. Anything you upload, download, or store as a single .mp4 still has one moov, and it is still in whichever place the encoder happened to put it.

What To Do About It

Always pass -movflags +faststart when producing MP4 for the web. It costs one extra pass over the file at encode time and nothing at all afterward.
Check where moov landed before blaming the network. Eight bytes of parsing tells you: read the size at offset 0, jump, read the type, repeat. If moov is last, that is your slow start.
Use fMP4 for anything live or adaptive. A single-file MP4 cannot be written and played at the same time, no matter how you order the boxes.
Don’t concatenate MP4 files. cat a.mp4 b.mp4 > c.mp4 produces a file whose first moov describes only the first video and whose second moov has offsets pointing into the wrong place. Remux with ffmpeg -f concat instead.
Treat .heic and .avif as the same problem space. If your image pipeline calls identify or sniffs magic bytes, those files start with a box header, not a signature, and the four bytes that matter are at offset 4 rather than offset 0.
Read ftyp compatible brands, not the extension. A file named .mp4 can declare qt , and a file named .mov can declare isom. The brands are the truth.

Everything in this series so far has failed by underspecifying something. MP4 does not have that problem. The box model is rigorous, self-describing, and general enough that it absorbed still images without anyone having to redesign it. What it got wrong was one thing: it wrote down byte offsets instead of relative ones, and made the index a single object that has to be complete before it can be written. Two decades of streaming infrastructure exist to work around that decision.

Sources

ISO/IEC 14496-12 — the ISO Base Media File Format; the box definition is clause 4.2
MP4 Registration Authority — the registry of every legal FourCC brand and box type
Apple’s QuickTime File Format documentation — the atom model MP4 inherited
AVIF specification — how AV1 intra frames map onto ISOBMFF items
RFC 8216 — HTTP Live Streaming, which is fMP4 plus a text manifest
ffmpeg movflags documentation — +faststart and the fragmentation options

Every XML Feature Is One You Turn Off

Sat, 22 Aug 2026 10:00:00 -0500

Last post ended on a question. XML is the format in this series that specified everything, so what did all that specifying buy?

An answer for every question, and an attack surface made entirely of answers.

The other formats in this series failed by leaving things out. CSV never had a standard. JSON declined to say what a number means. YAML wrote the rules down and then most implementations kept using the old ones. XML failed the other way. It defined a schema language, a query language, a transformation language, a namespace system, and an entity system, and the entity system will read files off your disk.

Two Kinds of Correct

Start with something XML got right, because it is the only format here that made this distinction at all.

The specification defines two separate bars:

Well-formed is syntax. Tags match, there is one root, attributes are quoted. Valid is semantics: the document declares a schema and conforms to it.

CSV has neither concept. JSON has only the first one. XML separated them in 1998, which was ahead of its time, and then almost nobody used the second one. Most XML in production is well-formed and unvalidated, which means it has exactly the same guarantees as JSON with more punctuation.

The Entity System

XML documents are built from entities. Five are predefined, and you know them:

&amp;  &lt;  &gt;  &quot;  &apos;

You can also declare your own in the document type declaration, which is how you get constants in a config file. And an entity can be declared to pull its content from somewhere else:

<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "file:///tmp/xmltest/secret.txt">
]>
<foo>&xxe;</foo>

That is a legal XML document. SYSTEM means “go get it.” Here is what a parser does with it:

ElementTree  ParseError: undefined entity &xxe;
lxml         XMLSyntaxError: Entity 'xxe' not defined
defusedxml   EntitiesForbidden(name='xxe', system_id='file:///tmp/xmltest/secret.txt')
lxml, resolve_entities=True    'SECRET-CANARY-12345\n'

The first three refuse. The fourth read a file off the disk and put its contents in the document tree.

The difference between the second line and the fourth is one keyword argument. Not a patch, not an old version, not a misconfiguration. A parser option, on a current library, named after something that sounds like ordinary XML processing. Of course you want entities resolved. Entities are how the format works.

This is XXE, and it is not a bug in any implementation. Every one of those parsers is behaving as specified. The specification says an external entity is fetched, so the ones that fetch it are correct and the ones that refuse are deliberately non-conforming for your safety.

The Bomb That Got Fixed

The same entity system nests, which produces the XML version of the billion laughs attack:

<!DOCTYPE lolz [
  <!ENTITY lol "lol">
  <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
  <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
]>

Each level multiplies by ten. A 291-byte file at three levels expands to 3,000 characters. Add two more levels and the arithmetic says 300,000, and the file is still under 500 bytes.

Except it doesn’t, and this is the part worth reporting honestly. At five levels, current libxml2 stops:

XMLSyntaxError: Maximum entity amplification factor exceeded,
see xmlCtxtSetMaxAmplification

It refuses even with resolve_entities=True. Somebody went and put a ratio limit in the parser, and it works. Compare that to YAML, where safe_load still expands aliases without complaint, and the equivalent 202-byte payload produced 74,732 nodes with no objection at all.

So the XML ecosystem fixed the denial-of-service and left the file-read one flag away. That ordering tells you something about which failure people actually hit.

What the Rest of It Bought

XML shipped an enormous amount of specification, and the pieces are individually good:

XSD defines 19 primitive datatypes with real inheritance, so <price>12.50</price> can be a decimal rather than a hopeful string. This is precisely what CSV lacks and what JSON refuses to commit to.
XPath addresses any node in a document without writing a traversal.
XSLT transforms one document into another declaratively.
Namespaces let two vocabularies coexist in one file without colliding, which is the problem every format in this series either ignores or solves with a naming convention.

None of that is bad engineering. Read the list again and notice that it is a description of the problems the other five posts were about. XML solved them. In 1998.

The cost was that using XML correctly means knowing which parts to switch off, and the defaults were set in an era that assumed documents came from people you knew. Every hardening guide for XML is a list of features to disable: no DTDs, no external entities, no network access, no schema fetching.

There is a version of this where the lesson is “XML was too complicated.” I don’t think that’s it. The formats that replaced it are simpler and they have the same problems, plus the ones XML had already solved. Nobody misses XSLT, and everybody has now written their own worse version of it.

What To Do About It

Disable DTD processing entirely unless you know you need it. In Python that is defusedxml; in Java it is disallow-doctype-decl. This closes XXE and entity expansion in one move.
Never enable resolve_entities on input you did not write. It is the single flag that turns a parser into a file reader.
Validate against a schema at the trust boundary, not just parse. Well-formed is not a security property. It is barely a correctness property.
Stream large documents. SAX and StAX read in constant memory; DOM builds the whole tree first, which is its own denial of service if the document is attacker-sized.
Know your parser’s defaults and pin them explicitly. They have changed over time, usually toward safety, and code that relies on a safe default is one dependency upgrade from a different one.

XML answered every question these posts have raised. It has types, a validation model, a query language, and a namespace system, and it had all of them a decade before the formats that replaced it. What it could not do was make the safe path the default one, and that turned out to matter more than any of the rest of it.

That is the actual pattern across this whole series. Not that formats are underspecified or overspecified, but that the defaults are the specification most people ever use.

Sources

XML 1.0 (Fifth Edition) — 26 November 2008; the well-formed and valid definitions, and the five predefined entities
XML 1.0 (First Edition) — 10 February 1998, the original Recommendation
XML 1.1 — 4 February 2004; the revision almost nobody adopted
XSD 1.1 Part 2: Datatypes — the 19 primitive datatypes
OWASP XXE Prevention Cheat Sheet — per-parser hardening settings
defusedxml — the Python library that turns the dangerous parts off for you

Norway Is Not a Boolean

Fri, 21 Aug 2026 10:00:00 -0500

JSON’s problem is that its specification is too small. It tells you 9007199254740993 is a well-formed number and then declines to say which number.

YAML went the other way. The 1.2.2 specification is a book. It has a formal grammar, a chapter on recommended schemas, and an answer for nearly everything. And it will still read your config file and decide, without asking, that Norway is false.

The Norway Problem

Here is a config file. Every value in it is a string that a human would read as a string.

country: NO
duration: 1:20
mode: 0755

Three parsers, on those exact bytes:

PyYAML 6.0.3   {'country': False, 'duration': 80,     'mode': 493}
ruamel.yaml    {'country': 'NO',  'duration': '1:20', 'mode': 755}
js-yaml        {"country":"NO",   "duration":"1:20",  "mode":755}

NO is the ISO 3166 code for Norway. PyYAML returns the boolean false, because YAML 1.1 recognized twenty-two spellings of true and false, and NO is one of them. The spec lists them as a single regular expression:

y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF

Count them. Twenty-two. Six of those are country codes, single letters, or ordinary English words that appear in real data. Nothing in the file said “this is a boolean.” The parser inferred it from the shape of the text, and the shape of the text was two letters.

1:20 became 80 because YAML 1.1 supported sexagesimal integers, so a duration is read as base 60. One times sixty, plus twenty.

0755 became 493 because a leading zero meant octal. That is a file mode that no longer means what it says.

It Was Fixed in 2009

This is the part that makes YAML different from the other formats in this series.

CSV never had a standard. Markdown had too many. YAML had exactly one problem, everybody agreed it was a problem, and the working group fixed it. YAML 1.2 arrived in 2009 and threw all of it out. Base 60 is gone. Implicit octal is gone. The Core schema recognizes true and false and their case variants, and nothing else.

Seventeen years later, the two YAML 1.2 parsers above return strings, and PyYAML returns False.

PyYAML implements YAML 1.1. It is the default YAML library for Python, it is what pip install pyyaml gives you, and the specification it implements was superseded when the iPhone 3GS was current. The fix exists. It shipped. Most of the ecosystem simply stayed where it was, because changing the type of NO in a minor release breaks every config file that relied on it.

A format can be fixed and still be broken, if the fix arrives after the implementations do.

Everything Else That Isn’t a String

The country-code case is famous. It is not the only one, and the rest are quieter:

version: 1.10   ->  1.1        (float, and .10 became .1)
build: 010      ->  8          (octal)
port: 8080      ->  8080       (int, fine, until you concatenate it)
answers: [y, n] ->  ['y', 'n'] (strings)
answers: [yes, no] -> [True, False]

The first one is the one that should bother you. A semantic version of 1.10 parses as the float 1.1, which is a different version, and it does it silently in a file whose entire job is to record which version you meant.

And note the last two lines. y and n stay strings in PyYAML while yes and no become booleans, because PyYAML’s resolver implements a narrower set than the 1.1 spec’s regexp advertises. So the answer to “does this parser coerce single letters” is neither yes nor no. It is “some of them, and you have to test.”

Two Ways to Weaponize the Convenience

YAML has anchors. You define a node once with &name and reference it with *name. It is a useful feature for config files with repeated blocks, and it composes.

That is the problem. It composes exponentially.

a: &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]

That file is 202 bytes. Expanding it produces 74,732 nodes, of which 59,049 are copies of the string lol. Add one more line and multiply by nine. This is the billion laughs attack, and the important detail is that safe_load does not stop it. Aliases are not a dangerous tag, they are a core language feature working as designed.

The second way is tags. YAML can annotate a node with a type, and PyYAML historically honored tags that construct arbitrary Python objects:

!!python/object/apply:os.system
args: ['id']

yaml.load() on untrusted input would run that. It became CVE-2017-18342, CVSS 9.8, published June 2018, with a description that is unusually blunt for the genre: “In PyYAML before 5.1, the yaml.load() API could execute arbitrary code if used with untrusted data.”

The fix took two releases and three years. PyYAML 5.1 deprecated the unsafe default in March 2019. PyYAML 6.0 finally made the Loader argument mandatory in October 2021, so the dangerous call stopped being the short one:

>>> yaml.load('a: 1')
TypeError: load() missing 1 required positional argument: 'Loader'

The vulnerability was published in 2018. Making the unsafe call harder to type than the safe one landed in 2021.

What To Do About It

Quote anything that isn’t obviously a number. Country codes, versions, file modes, git SHAs, anything a human would call an identifier. Quoting is never wrong.
Know which YAML version your parser speaks. If it is Python, assume 1.1 and the Norway problem unless you chose otherwise. ruamel.yaml gives you 1.2.
Never call yaml.load on input you did not write. safe_load, always. On PyYAML 6 the language makes you say which you meant, which is the correct design.
Bound the input. safe_load is not a defense against alias expansion. If you parse YAML you did not author, cap the document size before it reaches the parser.
Use a schema. The value of a schema here is not validation, it is that it declares the type instead of letting the parser guess it from the characters.

YAML’s failure is the opposite of JSON’s, and it produces the same result. JSON declined to say what values mean, so implementations disagreed. YAML said what values mean in enormous detail, got it wrong in 2005, corrected it in 2009, and the correction never fully landed.

Next in this series is XML, which is the one format here that did specify everything. It has a schema language, a query language, a transformation language, and a namespace system. It is worth asking what all of that bought.

Sources

YAML 1.2.2 Specification — October 2021; the schemas chapter and the rule that tabs “must not be used in indentation, since different systems treat tabs differently”
YAML 1.1 Boolean type — the twenty-two-form regexp, working draft dated 2005
CVE-2017-18342 — the yaml.load() RCE, CVSS 9.8
PyYAML CHANGES — 5.1 (2019) deprecated the unsafe default, 6.0 (2021) made Loader required

Your JSON Parser Disagrees With Mine

Thu, 20 Aug 2026 10:00:00 -0500

Markdown has too many specs. CSV has one that nobody agreed to follow. JSON is supposed to be the happy ending: a grammar small enough to print on a business card, standardized twice over, and elevated to a full Internet Standard.

It is, and the parsers still disagree with each other about what your file says.

Not about whether it’s valid. About what the values are.

The Same Number, Three Answers

Here is a JSON document. It is unambiguously valid by every specification.

{"id": 9007199254740993}

Three parsers, on the same machine (Node 24.13.0, Python 3.13.12, jq 1.8.2), on those exact bytes:

node   : {"id":9007199254740992}
python : {"id": 9007199254740993}
jq     : {"id":9007199254740993}

Node gave back a different number than the one in the file. It didn’t error, didn’t warn, didn’t round-trip. The last digit changed from 3 to 2.

The reason is that JSON’s grammar allows a number to be any sequence of digits, but JavaScript represents every number as an IEEE 754 double. Above 2⁵³ the integers stop being exactly representable, and Number.MAX_SAFE_INTEGER is 9007199254740991. Our value is two past it.

This is not a JavaScript bug. Node is behaving exactly as specified. The specification simply declines to say how many digits a number may have, and every implementation answers that question with whatever its host language does.

Those version numbers matter, which is its own version of the problem. jq only began preserving decimal literals in 1.7, whose release notes list “use decimal number literals to preserve precision.” Run that same file through jq 1.6 and it goes through a double and hands you Node’s answer. The tool doesn’t just disagree with other parsers. It disagrees with its own past self.

If you have ever wondered why APIs send 64-bit IDs as strings, this is why. Twitter’s snowflake IDs, database primary keys, anything above 2⁵³ has to be quoted or it silently degrades in half the ecosystem.

Duplicate Keys Are Legal

{"role": "user", "role": "admin"}

RFC 8259 says names within an object should be unique. Should, not must. And it goes on to describe what happens otherwise as varying between implementations.

In practice:

node   : {"role":"admin"}
python : {"role": "admin"}
jq     : {"role":"admin"}

All three take the last one. That’s the common behavior, and it is not required.

The RFC itself spells out all three possibilities:

That third option, keeping both, isn’t even representable in most languages’ object types. Nicolas Seriot tested parsers across a dozen languages against cases like this one and concluded there are “no two parsers that agree on what is wrong and what is right.”

Python will show you both if you ask:

raw pairs: [('role', 'user'), ('role', 'admin')]

The pairs are all there in the document. Choosing one is an interpretation layered on top of parsing.

Now put two parsers in one system. Apache CouchDB did, and it became CVE-2017-12635.

CouchDB used an Erlang parser for authentication and a JavaScript engine for the validation that runs when a document is written. The Erlang parser resolved duplicate keys to the first value. The JavaScript engine resolved them to the last. So a request like this:

{"roles": ["_admin"], ..., "roles": []}

was read by the write-time validation as an ordinary unprivileged user, because it saw the last roles key and found it empty. It was then read by the authentication layer as an administrator, because that saw the first one. Non-admin users could grant themselves admin.

CouchDB’s fix was to change the Erlang parser to take the last key, matching JavaScript. Not because last-wins is correct, but because agreeing is correct.

This is the JSON version of the ZIP two-index problem from a few posts back. When a format permits two answers to the same question, the gap between two components is where the vulnerability lives.

NaN Is Not JSON, and Python Emits It Anyway

JSON has no way to express not-a-number or infinity. The grammar has no room for them.

Python’s standard library writes them regardless:

>>> json.dumps({"a": float("nan"), "b": float("inf")})
'{"a": NaN, "b": Infinity}'

That output is not JSON. It’s Python’s default behavior, and it produces a file that other parsers reject or mangle. Feeding those exact bytes onward:

node : SyntaxError - Unexpected token 'N', "{"a": NaN, "b": "... is not valid JSON
jq   : {"a":null,"b":1.7976931348623157e+308}

Node’s response is correct and useful: this is not JSON, here’s where it broke.

jq’s response is the one that should worry you. It accepted the invalid document and made up values. NaN became null. Infinity became 1.7976931348623157e+308, the largest finite double. No error, no warning. If that ran in the middle of a data pipeline you would get numbers out the other end, and they would be wrong in a way no downstream check is likely to catch.

The same divergence shows up with a merely-enormous exponent, which is valid JSON:

input: {"v": 1e999}

node   : {"v":null}
python : {'v': inf}
jq     : {"v":1E+999}

Three parsers, one valid input, three different values. Node converts to infinity then serializes it as null because it can’t represent infinity on the way out. Python gives you a float infinity object. jq preserves the literal.

Why a Small Spec Doesn’t Save You

JSON’s specifications are good, and they are small. The problem is that they specify syntax, and almost every failure above is about semantics.

The grammar tells you 9007199254740993 is a well-formed number. It does not tell you what number it is, because that would require committing to a numeric model, and committing to a numeric model would have meant excluding some language from implementing JSON natively. The looseness is why JSON is everywhere. It is also why the same bytes mean different things in different places.

The standards process eventually acknowledged this. RFC 7493 defines I-JSON, a restricted profile that closes these holes: no duplicate names at all, numbers that should not exceed what an IEEE 754 double holds exactly, high-precision values recommended to travel as strings, and mandatory UTF-8. Only the duplicate-name rule is a hard MUST NOT, which tells you something about how much of this was still negotiable in 2015.

I-JSON is what most people think JSON already is. It exists as a separate document precisely because JSON isn’t that.

What To Do About It

Send large integers as strings. Anything that could exceed 2⁵³: IDs, timestamps in nanoseconds, financial values in minor units.
Reject duplicate keys at your trust boundary rather than letting your parser pick. Most libraries offer a hook.
Don’t let a language’s default serializer decide whether it emits valid JSON. Python needs allow_nan=False to be honest.
Validate before you transform. A parser that repairs invalid input is more dangerous than one that rejects it, because the repair is silent.
Target I-JSON for anything crossing a system boundary. It costs nothing and removes the whole category.

JSON did not fail. It succeeded so completely that it got implemented hundreds of times by people reading a short document, and a short document leaves a lot of decisions to the reader. The format that’s easy to implement is the format that gets implemented differently everywhere.

That’s the same sentence I could have written about Markdown, and about CSV. The pattern across this whole series is that a format’s ambiguities don’t stay theoretical. They become somebody’s incident.

Sources

RFC 8259 — the current JSON standard, and STD 90
RFC 7493 — I-JSON, the profile that closes the interoperability holes
ECMA-404 — the parallel Ecma grammar standard
Nicolas Seriot, “Parsing JSON is a Minefield” — the systematic survey of parser disagreement
JSONTestSuite — the executable test corpus behind that research, over 300 cases
CouchDB’s writeup of CVE-2017-12635 — the duplicate-key privilege escalation, in the vendor’s own words

Nobody Agrees What a CSV Is

Wed, 19 Aug 2026 10:00:00 -0500

CSV is simple but powerful. Values, separated by commas. It’s easy to understand and use.

It is also, by a wide margin, the one that destroys the most data.

That’s not a paradox. It’s cause and effect. A format simple enough that everyone writes their own parser is a format with as many dialects as it has parsers, and CSV’s defining property is that it carries no information about how to read it.

There Is No Standard

RFC 4180 exists. Yakov Shafranovich published it in October 2005, it registers the text/csv media type, and it gives an ABNF grammar.

It is also Informational, not Standards Track. It was written to describe what people were already doing, twenty-odd years after spreadsheets started emitting it. Section 2 says so outright: there is “no formal specification in existence,” and what follows documents “the format that seems to be followed by most implementations.”

By the time someone wrote it down, every spreadsheet, database, and scripting language had already shipped its own interpretation. The RFC didn’t settle anything. It just added one more dialect, with the distinction of having a number.

Eight years later, RFC 7111 added URI fragments for pointing at a row, column, or cell inside a text/csv file. Also Informational. CSV still had no standard, but you could now cite a specific cell of one.

Where Does a Record End?

The obvious answer is “at the newline,” and the obvious answer is wrong, because a quoted field is allowed to contain one.

name,notes
Alice,"line one
line two"
Bob,fine

That’s a valid three-row CSV. Split it on newlines and you get four:

naive split gives 4 lines:
   'name,notes'
   'Alice,"line one'
   'line two"'
   'Bob,fine'

a real CSV parser gives 3 rows:
   ['name', 'notes']
   ['Alice', 'line one\nline two']
   ['Bob', 'fine']

Every head, wc -l, split("\n"), and shell pipeline that assumes one record per line is wrong on this file. Not wrong on a malformed file. Wrong on a correct one.

This is the single most common CSV bug, and it’s invisible in testing, because your test fixtures don’t have newlines in them until a user pastes an address into a form.

What’s the Delimiter?

In most of Europe the decimal separator is a comma. 12,50 is twelve and a half euros. Which means a comma cannot also be a field separator, so those locales use semicolons.

Feed a German CSV to an RFC 4180 parser and everything survives, in the sense that nothing throws:

input:  produkt;preis
        Kaffee;12,50
        Tee;9,90

parsed as comma-delimited:
   ['produkt;preis']
   ['Kaffee;12', '50']
   ['Tee;9', '90']

parsed as semicolon-delimited:
   ['produkt', 'preis']
   ['Kaffee', '12,50']
   ['Tee', '9,90']

The first reading gives you two columns of nonsense with no error. The prices split down the middle of the decimal point. A pipeline that ingests this will happily compute statistics on the number 12 and the number 50.

Excel picks the delimiter based on your operating system’s regional settings, which means the same file opens differently on two machines in the same office.

Is the First Row a Header?

1,2,3
4,5,6

Header or data? Nothing in the file says.

RFC 4180’s answer is that you put it in the MIME type: text/csv; header=present. Which is a real answer, and also means the information lives outside the file, in a transport layer that gets stripped the moment someone saves the attachment to disk.

So in practice every tool guesses, usually by checking whether the first row looks less numeric than the rest.

The Part That Destroys Data

Everything above is a parsing problem. This one is worse, because the file parses fine and the damage happens after.

CSV has no types. Every value is text. So every spreadsheet and dataframe library applies type inference on import, and type inference is lossy.

Here’s a file with four columns of identifiers, all of which are strings that happen to be made of digits:

gene,zip,card,accession
SEPT7,02138,4532012345678901,0004928
MARCH1,01234,4111111111111111,0000071

Read it with a type-inferring reader and:

  gene  zip             card  accession
 SEPT7 2138 4532012345678901       4928
MARCH1 1234 4111111111111111         71

The ZIP code 02138 is now 2138. The accession number 0004928 is now 4928. Nobody was asked. Nothing warned. Save that back to CSV and the original values are gone from disk.

Spreadsheets are worse than this, because they store every number as an IEEE 754 double. Microsoft is blunt about the consequence:

The example they reach for is a card number:

typed into a cell     1234 5678 9087 6543
Excel shows           1.23E+15

Microsoft calls that “truncating numerical data to 15 digits of precision and converting to a number displayed in scientific notation.” Note that this is the vendor describing its own product, not a bug report.

The card number in the file above is also 16 digits. pandas read it back intact. Excel would not.

Credit card numbers are 16 digits. Many national ID numbers are longer. They are not numbers in any meaningful sense, they’re strings of digits, and a format with no type information cannot tell the difference.

The Gene Name Problem

The best-documented case of this is genomics, because biologists name genes things like SEPT1 and MARCH1 and spreadsheets read those as dates.

In 2016 Ziemann and colleagues screened 35,175 supplementary Excel files from 18 journals covering 2005 to 2015. Among articles containing Excel gene lists, 19.6% had gene names corrupted this way. One in five.

A follow-up in 2021, “Gene name errors: Lessons not learned,” found 30.9% across a broader sample drawn from PubMed Central. Worth being careful comparing those two numbers directly, because the second study used a different sampling frame and also detected an additional error category the first one didn’t look for. The honest summary is that the problem did not go away in the five years after being loudly published.

The resolution is the remarkable part. The field did not fix the spreadsheets. It renamed the genes. The HUGO Gene Nomenclature Committee’s 2020 guidelines state that “all symbols that auto-converted to dates in Microsoft Excel have been changed,” giving SEPT1 becoming SEPTIN1 and MARCH1 becoming MARCHF1 as examples.

Human genes were renamed because a file format cannot say what type a column is.

What To Do About It

CSV isn’t going away, and mostly shouldn’t. It’s readable, streamable, diffable, and every tool on earth reads it.

The practical defenses are short:

Quote everything. It’s never wrong and it removes a whole class of ambiguity.
Treat identifiers as strings explicitly at the point of import. Every serious CSV reader lets you pin column types; use it.
Never round-trip through a spreadsheet if the data contains identifiers. Opening and saving is a lossy operation.
Say what you mean out of band. Delimiter, encoding, header presence, quoting style. The file will not.
Use something else when you can. Parquet and even JSON Lines carry types. If the consumer is a program rather than a person, the readability argument for CSV mostly evaporates.

The lesson generalizes past CSV, and it’s the same one from the text file post. A format that carries no description of itself pushes that burden onto every reader, and readers guess. Usually well. Occasionally by silently deleting the leading zero from your ZIP code.

Sources

RFC 4180 — the Informational spec that documents CSV rather than defining it
RFC 7111 — URI fragment selectors for text/csv, January 2014
Ziemann et al., “Gene name errors are widespread in the scientific literature” — Genome Biology, 2016; the 19.6% figure (free full text)
Abeysooriya et al., “Gene name errors: Lessons not learned” — PLOS Computational Biology, 2021; the 30.9% follow-up
Bruford et al., “Guidelines for human gene nomenclature” — Nature Genetics, 2020; the renaming (free full text)
Microsoft on Excel’s floating-point precision — Excel follows IEEE 754 and stores 15 digits of precision
Microsoft on leading zeros and large numbers — “any numbers past the 15th digit are rounded down to zero,” with a credit card as the example
Microsoft on importing and exporting text files — the CSV list separator comes from Windows Region settings

Nobody Agrees What a CSV Is

Tue, 18 Aug 2026 10:00:00 -0500

CSV is the simple but powerful. Values, separated by commas. It’s easy to understand and use.

It is also, by a wide margin, the one that destroys the most data.

There Is No Standard

RFC 4180 exists. Yakov Shafranovich published it in October 2005, it registers the text/csv media type, and it gives an ABNF grammar.

Where Does a Record End?

The obvious answer is “at the newline,” and the obvious answer is wrong, because a quoted field is allowed to contain one.

name,notes
Alice,"line one
line two"
Bob,fine

That’s a valid three-row CSV. Split it on newlines and you get four:

naive split gives 4 lines:
   'name,notes'
   'Alice,"line one'
   'line two"'
   'Bob,fine'

a real CSV parser gives 3 rows:
   ['name', 'notes']
   ['Alice', 'line one\nline two']
   ['Bob', 'fine']

Every head, wc -l, split("\n"), and shell pipeline that assumes one record per line is wrong on this file. Not wrong on a malformed file. Wrong on a correct one.

This is the single most common CSV bug, and it’s invisible in testing, because your test fixtures don’t have newlines in them until a user pastes an address into a form.

What’s the Delimiter?

In most of Europe the decimal separator is a comma. 12,50 is twelve and a half euros. Which means a comma cannot also be a field separator, so those locales use semicolons.

Feed a German CSV to an RFC 4180 parser and everything survives, in the sense that nothing throws:

input:  produkt;preis
        Kaffee;12,50
        Tee;9,90

parsed as comma-delimited:
   ['produkt;preis']
   ['Kaffee;12', '50']
   ['Tee;9', '90']

parsed as semicolon-delimited:
   ['produkt', 'preis']
   ['Kaffee', '12,50']
   ['Tee', '9,90']

Excel picks the delimiter based on your operating system’s regional settings, which means the same file opens differently on two machines in the same office.

Is the First Row a Header?

1,2,3
4,5,6

Header or data? Nothing in the file says.

So in practice every tool guesses, usually by checking whether the first row looks less numeric than the rest.

The Part That Destroys Data

Everything above is a parsing problem. This one is worse, because the file parses fine and the damage happens after.

CSV has no types. Every value is text. So every spreadsheet and dataframe library applies type inference on import, and type inference is lossy.

Here’s a file with four columns of identifiers, all of which are strings that happen to be made of digits:

gene,zip,card,accession
SEPT7,02138,4532012345678901,0004928
MARCH1,01234,4111111111111111,0000071

Read it with a type-inferring reader and:

  gene  zip             card  accession
 SEPT7 2138 4532012345678901       4928
MARCH1 1234 4111111111111111         71

The ZIP code 02138 is now 2138. The accession number 0004928 is now 4928. Nobody was asked. Nothing warned. Save that back to CSV and the original values are gone from disk.

Spreadsheets are worse than this, because they store every number as an IEEE 754 double. Microsoft is blunt about the consequence:

The example they reach for is a card number:

typed into a cell     1234 5678 9087 6543
Excel shows           1.23E+15

The card number in the file above is also 16 digits. pandas read it back intact. Excel would not.

The Gene Name Problem

The best-documented case of this is genomics, because biologists name genes things like SEPT1 and MARCH1 and spreadsheets read those as dates.

Human genes were renamed because a file format cannot say what type a column is.

What To Do About It

CSV isn’t going away, and mostly shouldn’t. It’s readable, streamable, diffable, and every tool on earth reads it.

The practical defenses are short:

Quote everything. It’s never wrong and it removes a whole class of ambiguity.
Treat identifiers as strings explicitly at the point of import. Every serious CSV reader lets you pin column types; use it.
Never round-trip through a spreadsheet if the data contains identifiers. Opening and saving is a lossy operation.
Say what you mean out of band. Delimiter, encoding, header presence, quoting style. The file will not.
Use something else when you can. Parquet and even JSON Lines carry types. If the consumer is a program rather than a person, the readability argument for CSV mostly evaporates.

Sources

RFC 4180 — the Informational spec that documents CSV rather than defining it
RFC 7111 — URI fragment selectors for text/csv, January 2014
Ziemann et al., “Gene name errors are widespread in the scientific literature” — Genome Biology, 2016; the 19.6% figure (free full text)
Abeysooriya et al., “Gene name errors: Lessons not learned” — PLOS Computational Biology, 2021; the 30.9% follow-up
Bruford et al., “Guidelines for human gene nomenclature” — Nature Genetics, 2020; the renaming (free full text)
Microsoft on Excel’s floating-point precision — Excel follows IEEE 754 and stores 15 digits of precision
Microsoft on leading zeros and large numbers — “any numbers past the 15th digit are rounded down to zero,” with a credit card as the example
Microsoft on importing and exporting text files — the CSV list separator comes from Windows Region settings

Word Documents Used to Be Filesystems

Tue, 18 Aug 2026 10:00:00 -0500

Last post ended on a promise: a .docx is a ZIP file, and you already know how ZIP works.

That’s true, and it’s the smaller half of the story. The interesting part is what .docx replaced, because the old .doc format was doing something strange. It wasn’t a document. It was a filesystem with a document living inside it.

A Filesystem in a File

Here’s a real .doc from 2015, 10,240 bytes. The first eight bytes:

d0 cf 11 e0 a1 b1 1a e1

That’s the Compound File Binary Format signature, also called OLE2. file(1) recognizes it and doesn’t even mention Word:

$ file "rich text.doc"
Composite Document File V2 Document, Little Endian, Os: Windows,
Version 1.0, Code page: -535, Revision Number: 0,
Create Time/Date: Thu Dec 10 13:38:22 2015

“Composite Document File” is the honest description. CFBF is a container that implements directories and files, called storages and streams, inside a single flat file. It has a File Allocation Table. It has sectors. If that sounds like FAT16, that’s because it’s the same idea, scaled down to live inside one file on a real filesystem.

Cracking this one open gives:

     106 bytes  CompObj
      20 bytes  Ole
     116 bytes  DocumentSummaryInformation
     312 bytes  SummaryInformation
    2411 bytes  1Table
    3620 bytes  WordDocument

sector size      : 2^9 = 512 bytes
mini sector size : 2^6 = 64 bytes

Two sector sizes, because a 512-byte sector is wasteful for a 20-byte stream. Streams under 4,096 bytes get allocated out of a separate mini-FAT in 64-byte units. There is a fragmentation strategy inside your Word document.

The WordDocument stream is the main event, and it opens with a File Information Block whose magic number is 0xA5EC:

WordDocument stream: 3620 bytes
  FIB magic (wIdent) = 0xA5EC

None of this is the text yet. This is all container.

The Text Is Not in Order

You’d expect the document’s text to sit in the WordDocument stream in reading order. It doesn’t. It sits there in edit order, and a separate structure called a piece table says how to reassemble it.

The piece table is a list of descriptors, each saying “characters at logical position X through Y live at physical offset Z.” Reading a .doc means walking that table and gathering fragments scattered through the stream.

Why build it that way? Because of a feature called Fast Save, and because in 1990 writing to disk was slow. When you edited a document, Word didn’t rewrite the file. It appended your new text to the end of the stream and updated the piece table to point at it. Saving a one-word change to a 200-page document meant writing a few dozen bytes instead of a few hundred kilobytes.

That’s a good optimization. It has an obvious and terrible consequence.

The old text is still in the file. Deleting a paragraph removed it from the piece table, not from the stream. The bytes stayed exactly where they were, unreferenced, invisible in Word, and completely readable in a hex editor.

Microsoft documented this themselves, in a knowledge base article about minimizing metadata in Word documents: “Because of the design of the FastSave feature, text that you delete from a document may remain in the document, even after you save the document.” The recommended fix was to go into Options and clear the “Allow fast saves” check box. From Word 97 SR-1 onward they turned it off by default.

For years, “open the document in a text editor and scroll” was a functioning technique for reading text someone believed they had deleted. Every organization circulating Word files was potentially shipping its own edit history.

The piece table itself has a respectable pedigree. Charles Simonyi brought the technique to Microsoft from Xerox PARC’s Bravo editor, and it’s an elegant way to represent an editable buffer. It’s still how many text editors model documents in memory. The mistake wasn’t the data structure. The mistake was persisting the whole scratch buffer to disk and shipping it to other people.

Then It Became a ZIP of XML

Office 2007 replaced all of it with the Open Packaging Conventions: ECMA-376, later ISO/IEC 29500. A .docx is a ZIP archive containing XML.

Every .docx opens with the same four bytes:

50 4b 03 04    <- PK\x03\x04, a ZIP local file header

PK. Phil Katz’s initials, from the last post, sitting at byte zero of every Word document written since 2007.

Unzip one and the structure is legible:

[Content_Types].xml
_rels/.rels
word/document.xml
word/_rels/document.xml.rels
word/styles.xml
word/settings.xml
word/fontTable.xml
word/theme/theme1.xml
docProps/core.xml
docProps/app.xml

word/document.xml holds the text. [Content_Types].xml maps each part to a MIME type. _rels/.rels is a relationship graph saying which part is the main document and how the parts connect. The whole thing is a tiny website, zipped.

The text itself is WordprocessingML:

<w:p>
  <w:r>
    <w:t>Hello, World!</w:t>
  </w:r>
</w:p>

A paragraph containing a run containing text. Verbose, but you can read it, and more importantly a program you wrote in an afternoon can read it. That is important when building foundational file formats that outlive the creators.

Extracting text from a .doc meant implementing a filesystem and a piece table. Extracting text from a .docx means unzipping and finding <w:t> elements.

The XML contains the document, not the document’s history. Deleted text is deleted.

XML Did Not Mean Simple

It would be tidy to end on “and then it got clean.” The specification runs to several thousand pages, and the ISO fast-track that pushed it through in 2008 was contentious enough to deserve its own post.

What matters here is the shape it settled into. The standard shipped split in two: Strict, the clean format, and Transitional, which carries the legacy baggage forward so documents converted from the binary era still render correctly.

Guess which one nearly everything emits.

Open a Transitional document’s settings and you find a <w:compat> block. Its children are a museum:

w:truncateFontHeightsLikeWP6    WordPerfect 6
w:suppressTopSpacingWP          WordPerfect
w:lineWrapLikeWord6             Word 6
w:autoSpaceLikeWord95           Word 95
w:footnoteLayoutLikeWW8         Word 97
w:useWord97LineBreakRules       Word 97
w:mwSmallCaps                   Mac Word

Every one of those is a flag asking the renderer to reproduce how a specific piece of 1990s software behaved. Not what the format should do. What Word 6 did do, quirks included. Implementing this correctly means emulating applications whose behavior was never written down anywhere.

The bugs were load-bearing, so they got standardized. The format stopped being a filesystem, but it did not stop being a thirty-year-old application’s memory dumped to disk. It just picked a more legible way to write it down.

Which is, in fairness, an enormous improvement. You can read the file now. You just can’t read all of it quickly.

Sources

MS-CFB: Compound File Binary Format — Microsoft’s spec for the OLE2 container
MS-DOC: Word Binary File Format — the FIB, the piece table, and the stream layout
ECMA-376 — Office Open XML, the basis for .docx, and free to download. This is the same specification ISO published as ISO/IEC 29500, so read it here rather than paying ISO for the identical text
Library of Congress format description for OOXML — preservation notes and format history
KB Q223790: WD97: How to Minimize Metadata in Word Documents — the fast-save warning, archived; Microsoft no longer hosts it
w:compat schema reference — the full list of compatibility settings, browsable without downloading the spec

ZIP Files Are Read Backwards

Mon, 17 Aug 2026 10:00:00 -0500

Every format in this series so far reads front to back. PNG starts with a signature and you walk chunks in order. A text file is bytes from the beginning. Markdown parsers scan line by line, top to bottom.

ZIP reads backwards. The index is at the end of the file, and a reader is expected to seek to the end first and work its way back.

That one decision explains almost everything strange about ZIP, including a few things that look like bugs and one thing that is definitely a bug.

The Index Lives at the End

Here’s a real ZIP containing two small text files. 241 bytes total. Scanning it for the four-byte record signatures gives the whole layout:

offset  bytes         ascii   record
     0  50 4b 03 04   P K . . Local File Header      <- hello.txt
    53  50 4b 03 04   P K . . Local File Header      <- second.txt
   108  50 4b 01 02   P K . . Central Directory Header
   163  50 4b 01 02   P K . . Central Directory Header
   219  50 4b 05 06   P K . . End of Central Directory

Every one of those starts with the same two bytes. 0x50 is decimal 80, which is P in ASCII. 0x4b is decimal 75, which is K.

PK. Phil Katz, who wrote PKZIP in 1989, put his initials in the first two bytes of every structure in the format, and they are still there in every .docx, .jar, and .epub on your machine.

The two bytes after PK are the record type: 03 04 for a local file header, 01 02 for a central directory entry, 05 06 for the end-of-central-directory record. Those aren’t printable characters, which is deliberate. A four-byte constant made of two readable letters and two control bytes is unlikely to appear by accident in text, and easy to spot by eye in a hex dump.

The last 22 bytes are the End of Central Directory record, and it’s the entry point:

50 4b 05 06 00 00 00 00 02 00 02 00 6f 00 00 00 6c 00 00 00 00 00

signature            0x06054b50
total CD records     2
central dir size     111 bytes
central dir offset   108
comment length       0

A reader opens the file, jumps to the end, finds that record, reads “the index is at offset 108,” seeks there, and reads the catalog. Listing the contents of a 4 GB archive touches a few hundred bytes.

Note that each file appears twice: once as a Local File Header immediately before its compressed data, and once as an entry in the Central Directory at the end. Hold that thought.

Why Would You Do This?

Because in 1989 you were writing to a floppy disk, and often to a floppy disk that wasn’t big enough.

If the index goes at the front, you have to know everything about every file before you write the first byte: how many files, how big each one compresses to, where each one lands. That means compressing everything to a temporary location, then writing the header, then copying it all back. On a machine with 640K of RAM and two floppy drives, that’s brutal.

Put the index at the end and you can stream. Compress a file, write it, remember where it went. Compress the next one. When you run out of files, write down everything you remembered. One pass, no temporary copy, and you never needed to know the total size in advance.

TAR solved the same problem by having no index at all, which is why tar has to read an entire archive to find one file, and why you cannot randomly access a .tar.gz. ZIP got both streaming writes and random-access reads. That’s the trade that made it win.

The Backwards Scan Is Fuzzier Than It Sounds

The EOCD record is 22 bytes, so you’d think a reader could just read the last 22 bytes and be done.

It can’t, because the record ends with a variable-length archive comment of up to 65,535 bytes. The signature isn’t at a fixed offset from the end of the file. So a reader has to seek near the end and scan backwards looking for the four-byte signature, potentially across 65,557 bytes.

Searching for a magic number is not the same as knowing where a structure is. If those four bytes happen to appear inside the comment, or inside compressed data near the end of the file, a naive parser can lock onto the wrong one. Different implementations pick different candidates when there’s more than one. This is a recurring source of “this archive opens in one tool and not another.”

You Can Put Anything in Front of a ZIP

If a reader finds the archive by scanning backwards from the end, then whatever sits at the front of the file is not the reader’s problem.

Take a valid 69-byte PNG, take the 241-byte ZIP, and concatenate them with cat. No special tooling:

$ file polyglot.png
polyglot.png: PNG image data, 1 x 1, 8-bit/color RGB, non-interlaced

$ unzip -l polyglot.png
  Length      Date    Time    Name
---------  ---------- -----   ----
       12  08-10-2026 16:44   hello.txt
       13  08-10-2026 16:44   second.txt
---------                     -------
       25                     2 files

One 310-byte file. An image viewer reads the PNG signature at byte 0 and renders an image. An archive tool scans backwards, finds the EOCD, and extracts two files. Both are correct. Neither is being fooled by a trick; they’re each doing exactly what their format says to do.

This is the mechanism behind self-extracting archives, where the front of the file is a real executable and the back is a real ZIP. The same property is why “GIFAR” attacks worked: a file that a server accepted as a harmless image was loaded by Java as an archive of classes.

It also means the offsets inside the Central Directory are relative to the start of the archive, not the start of the file, and readers have to work out that difference. Prepending data shifts everything, and well-behaved parsers cope by computing the delta between where the EOCD says the directory should be and where it found it.

Two Indexes, One File

Back to that detail from earlier: every file’s name and metadata are stored twice, in the Local File Header and again in the Central Directory.

Nothing enforces that they agree.

Here’s the same archive with only the Central Directory copy of the first filename patched from hello.txt to BOGUS.txt. The local header is untouched:

$ unzip -l mismatch.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       12  08-10-2026 16:44   BOGUS.txt
       13  08-10-2026 16:44   second.txt

  local file header at offset 0 still says: hello.txt

The archive is not corrupt. unzip lists it happily. It just contains two different answers to “what is this file called,” and which one you get depends on which structure your parser decided to trust.

Now imagine two programs reading the same archive, one checking a signature and the other extracting files. That’s the Android “Master Key” bug from 2013, and the detail is better than the summary.

An APK is a ZIP. The attacker puts two entries in it, both named classes.dex. Android’s Java verifier loaded entries into a map keyed by filename, so a duplicate name overwrote the earlier one and the last entry was the one whose signature got checked. The native installer used a hash table with linear probing that didn’t replace on collision, so the first entry was the one that got loaded and run. Plant malicious code first, legitimately signed code second, and the device verifies one file and executes the other.

A second bug the same year came from the same “two readings, one file” family, via a signed integer. The extra-field length is a 16-bit value, and the Java code read it signed. A length of 65,533 (0xFFFD) sign-extends to −3. Since the offset of the compressed data is computed by adding that length, a negative value moves the read pointer backward into the header region instead of forward past it.

The lesson generalizes past ZIP. Any format that stores the same fact twice has to decide what happens when the copies disagree, and “the spec doesn’t say” is the same answer as “attackers decide.”

Offsets Are Just Numbers

The Central Directory locates each file by offset. Nothing in the format says two entries can’t point at the same bytes.

The classic zip bomb didn’t need that. 42.zip is 42 kilobytes of archives nested five layers deep, sixteen at each layer, unpacking to roughly 4.5 petabytes. The defense is obvious once you’ve seen it: cap recursion depth, don’t auto-extract nested archives.

David Fifield’s 2019 construction doesn’t recurse at all. It expands in a single pass, so depth limits are irrelevant. The trick is overlap: many Central Directory entries reference one shared kernel of compressed data, and each entry’s compressed stream uses DEFLATE’s stored-block mode to quote the next entry’s local file header as literal bytes. Entries nest inside each other, and output grows quadratically against input.

He published several, and they aren’t interchangeable:

File	Compressed	Uncompressed	Ratio	Needs Zip64
`zbsm.zip`	42 KB	5.5 GB	~130,000:1	No
`zblg.zip`	10 MB	281.4 TB	~28,000,000:1	No
`zbxl.zip`	46 MB	4.5 PB	~98,000,000:1	Yes

The Zip64 requirement on the largest one matters, because not every reader supports Zip64, which makes the merely-enormous version the more portable weapon.

None of these are malformed files. Every one is a valid archive that a conforming parser is supposed to accept. The format allows two entries to describe the same bytes, and no rule anywhere says the total uncompressed size has to bear any relationship to the file you’re holding.

Everything Is Secretly a ZIP

Once you know the structure, you start recognizing it:

.docx, .xlsx, .pptx are ZIP archives of XML
.jar, .war, .apk are ZIP archives of class files and resources
.epub is a ZIP of XHTML
.odt, .ods are ZIP of XML again

That’s not a coincidence or a hack. ISO/IEC 21320-1, “Document Container File,” defines a constrained ZIP profile for exactly this use. It narrows the format so a .docx reader doesn’t have to implement all of ZIP’s accumulated history: compression must be stored or deflated and nothing else, and the various encryption and digital-signature mechanisms in the original spec are all forbidden.

It’s a narrowing, not a rewrite. Zip64 version 1 is still permitted, for instance; only version 2 is ruled out. The profile is best understood as a list of the parts of ZIP that turned out to be a bad idea.

Which means the next post in this series is mostly about a ZIP file with XML inside it. You already know half of how a Word document works.

Sources

PKWARE APPNOTE.TXT — the original and still-authoritative ZIP specification, currently version 6.3.10; §4.3.16 defines the end of central directory record
ISO/IEC 21320-1:2015 — the constrained ZIP profile used by document formats. Fair warning, this one is a paid ISO standard; the catalog page tells you what it covers but you cannot read the text without buying it
Library of Congress format description for ZIP — history and preservation notes
David Fifield: A Better Zip Bomb — the overlapping-stream construction

Markdown Is Not a Format, It's an Argument

Sun, 16 Aug 2026 10:00:00 -0500

I’ve covered PNG and text files, and now it’s time for Markdown, which can be thought of as a philosophy of formatting or a lifestyle of text documents more so than an actually well defined file format. It has structure, and it has specifications, plural, and nothing agrees.

Here is three lines of Markdown run through five parsers:

INPUT: "- outer\n  - inner\n"

Python-Markdown      <ul> <li>outer</li> <li>inner</li> </ul>
markdown2            <ul> <li>outer <ul> <li>inner</li> </ul></li> </ul>
mistune              <ul> <li>outer<ul> <li>inner</li> </ul> </li> </ul>
marko (CommonMark)   <ul> <li> outer<ul> <li>inner</li> </ul> </li> </ul>
cmark-gfm (GitHub)   <ul> <li>outer <ul> <li>inner</li> </ul> </li> </ul>

Five parsers, five different results. Most of that is cosmetic whitespace, but look at the first one: Python-Markdown produced a flat list. The nesting is gone. That’s not a formatting difference, that’s a different document.

The Original Spec Was an Essay

John Gruber released Markdown in March 2004, along with a Perl script called Markdown.pl. The design goal was stated plainly:

That goal was met, and it’s why we’re all still using it twenty years later. The syntax borrowed from conventions people had already invented for plain text email and Usenet: = and - underlines from Setext, # headers from atx, > quoting from Usenet, * for emphasis from Textile and reStructuredText. None of it was new. That was the point.

What Markdown shipped without was a grammar. The specification was English prose describing the syntax with examples, and the tiebreaker for anything the prose didn’t cover was “whatever Markdown.pl does.” A Perl script full of regular expressions became the definition of the format by default.

That works fine until someone writes a second implementation.

Where the Prose Ran Out

The ambiguities weren’t exotic. They were things you hit in the first week:

How much indentation nests a list? Two spaces? Four? One tab? The original prose didn’t say clearly, and the answer interacts with the rule that four spaces means a code block.

What happens inside raw HTML? If you write a <div> and put Markdown inside it, does the Markdown get processed? Gruber’s implementation had behavior; the prose didn’t specify it.

When does a * open emphasis versus just being an asterisk? In a * b * c, are those multiplication signs or emphasis delimiters?

Do underscores work inside words? This one bites daily:

INPUT: "snake_case_variable"

Python-Markdown      <p>snake_case_variable</p>
markdown2            <p>snake<em>case</em>variable</p>
mistune              <p>snake_case_variable</p>
marko (CommonMark)   <p>snake_case_variable</p>
cmark-gfm (GitHub)   <p>snake_case_variable</p>

markdown2 italicizes your variable name. Every other parser leaves it alone. Both are defensible readings of a spec that never addressed it.

Or the heading with no space after the hash:

INPUT: "#Heading"

Python-Markdown      <h1>Heading</h1>
markdown2            <h1>Heading</h1>
mistune              <p>#Heading</p>
marko (CommonMark)   <p>#Heading</p>
cmark-gfm (GitHub)   <p>#Heading</p>

Half of them give you a heading, half give you a paragraph starting with a hash. This one matters because #hashtag at the start of a line is a real thing people write.

Everyone Wrote Their Own

With no formal spec, every implementation became a dialect, and the popular ones added features:

PHP Markdown Extra (Michel Fortin, 2005) added pipe tables, definition lists, footnotes, fenced code blocks, and attribute blocks.
MultiMarkdown (Fletcher Penney, 2005) added metadata frontmatter, cross-references, citations, and LaTeX export.
Pandoc Markdown (John MacFarlane, 2006) built a real AST-based parser and added YAML frontmatter, TeX math, grid tables, and citations.
kramdown (Thomas Leitner, 2009) added inline attribute lists and its own math support.

Each is a superset of a slightly different reading of the original. A document written for one is not guaranteed to render correctly in another, and the failure mode is silent: you don’t get a parse error, you get the wrong document.

CommonMark: Specify the Ambiguity Away

On 3 September 2014, Jeff Atwood announced a spec effort on Coding Horror under the name Standard Markdown, with John MacFarlane as primary author and people from GitHub, Reddit, Stack Exchange, and Meteor involved. The goal was not a new dialect and not a replacement for Gruber’s syntax, but an unambiguous description of what the existing syntax should mean in every case.

The name lasted about a day. That night, by Atwood’s account, Gruber emailed him and MacFarlane privately, called the name “infuriating,” and asked that the project be renamed and the domain taken down. On 4 September, Atwood published a follow-up retitling it Common Markdown, which shortly became the one-word CommonMark.

Worth being precise here, because this story gets retold badly: this was not a trademark action. Gruber holds no registered trademark on “Markdown” and did not invoke one. It was an objection to the name, made in private email, and the only public record of his side is Atwood’s paraphrase. There is no Daring Fireball post about it.

The naming fight is a footnote. The approach is the interesting part. Rather than describing the syntax in prose and hoping, CommonMark defines a parsing algorithm and ships an executable test suite pairing exact input with exact expected HTML, more than 500 examples embedded in the spec document itself. Conformance is not a matter of opinion. You run the tests.

The algorithm works in two passes.

Phase one walks the document line by line and builds block structure. Container blocks (blockquotes, lists, list items) and leaf blocks (headings, code blocks, paragraphs, HTML blocks) get assembled into a tree. Link reference definitions get collected. No inline formatting is considered at all in this phase, which is why block structure always wins: a > at the start of a line is a blockquote marker regardless of what emphasis you thought you were in the middle of.

Phase two walks the text inside leaf blocks and resolves inline structure. This is where emphasis, links, images, code spans, and inline HTML get parsed, using a delimiter stack.

That two-phase split is the single most useful thing to know about Markdown parsing, because it explains most surprising behavior. If your emphasis “leaked” across a list item boundary, it didn’t; blocks were decided before emphasis was ever considered.

The Emphasis Rules Are Hard

Emphasis is the hardest part of the spec, and CommonMark’s solution is a set of flanking rules. A run of * or _ is classified as left-flanking (can open emphasis) or right-flanking (can close it) based on the characters on either side, roughly: a delimiter can open if it’s not followed by whitespace, and can close if it’s not preceded by whitespace, with extra conditions around punctuation.

Then there’s a special case for underscores: an _ can open emphasis only if it’s left-flanking and not right-flanking. That single asymmetry is what makes snake_case_variable safe, because the middle underscores are both left- and right-flanking and are therefore disqualified from opening anything. Asterisks don’t get that rule, which is why snake*case*variable still italicizes.

This is what “specifying the ambiguity away” costs. The rule isn’t elegant. It exists because real documents contain identifiers, and a spec that italicizes your variable names is wrong no matter how clean its grammar is.

You can see the payoff in the nesting case:

INPUT: "*foo**bar**baz*"

Python-Markdown      <p><em>foo</em><em>bar</em><em>baz</em></p>
everyone else        <p><em>foo<strong>bar</strong>baz</em></p>

Four parsers agree, and the one that predates the delimiter-stack approach gets it wrong in a way that changes the meaning.

GFM Is a Layer, Not a Fork

GitHub Flavored Markdown is CommonMark plus five extensions, and it’s specified against CommonMark rather than diverging from it:

Tables, pipe-delimited with alignment colons
Task lists, - [ ] and - [x], rendered as checkboxes
Strikethrough, ~~text~~
Autolinks, bare URLs linkified without brackets
A raw HTML filter that neutralizes dangerous tags by escaping their opening bracket

That last one is a security control rather than a formatting feature, which tells you something about what it’s like to run a Markdown renderer on user-submitted content at GitHub’s scale.

The extension boundary is visible if you feed the same table to both:

INPUT:
| a | b |
|---|---|
| 1 | 2 |

CommonMark  <p>| a | b | |---|---| | 1 | 2 |</p>
cmark-gfm   <table><thead><tr><th>a</th><th>b</th></tr></thead>...

Tables are not Markdown. Tables are a GFM extension. CommonMark renders that input as a paragraph containing literal pipe characters, and it is correct to do so.

Tables, footnotes, task lists, strikethrough, frontmatter, math, and Mermaid diagrams are all extensions. None of them are guaranteed anywhere.

What To Do About It

The practical takeaways are short.

Know which parser you’re targeting. “It renders on GitHub” tells you about cmark-gfm, and nothing about your static site generator, your docs pipeline, or someone’s RSS reader.

Prefer the constructs everyone agrees on. Headings with a space after the hash, fenced code blocks, asterisks for emphasis, blank lines between blocks, four-space or consistent nesting. Boring Markdown survives transport.

Don’t rely on parser-specific behavior you discovered by accident. If nesting a list at two spaces works in your tool, that’s your tool, not the format.

There is even a formal way to say which dialect you mean. RFC 7763 registers text/markdown as a media type, and RFC 7764 defines a variant parameter for exactly this problem:

text/markdown; variant=CommonMark
text/markdown; variant=GFM
text/markdown; variant=Original

The standards process looked at Markdown, concluded that saying “this is Markdown” is not specific enough to be useful, and standardized a way to say which Markdown you meant.

That’s the tradeoff Markdown made. PNG picked one answer and enforced it with a checksum. A text file refuses to answer anything. Markdown let a million answers bloom, got adopted everywhere precisely because it was easy to implement badly, and has spent the last decade trying to agree with itself.

I’ll take that trade. But it’s worth knowing that when you write Markdown, you are not writing in a format. You’re writing in a dialect, and hoping the reader speaks it.

Sources

Daring Fireball: Markdown — Gruber’s original 2004 syntax document and design goals
CommonMark Specification — the parsing algorithm, emphasis flanking rules, and executable test suite
CommonMark parsing strategy appendix — the two-phase block/inline design
GitHub Flavored Markdown Spec — the five extensions, specified against CommonMark
RFC 7763 and RFC 7764 — the text/markdown media type and the registered dialect variants, both by S. Leonard, March 2016
Coding Horror: Standard Flavored Markdown and Standard Markdown is now Common Markdown — Atwood’s announcement and the rename a day later
Daring Fireball: Introducing Markdown — the original 15 March 2004 announcement
tagfilter.c in cmark-gfm — the nine tags GFM’s raw HTML filter neutralizes

There Is No Such Thing as a Text File

Sat, 15 Aug 2026 10:00:00 -0500

Last time I took apart PNG, which opens with eight bytes whose entire job is to announce “I am a PNG”.

A text file opens with nothing. No signature, no header, no length field, no version, no metadata. It is bytes, and then it stops.

So this post is the opposite of the last one. Instead of walking a structure, we’re going to look at what happens when there isn’t one.

POSIX Defines the Standard

Start with the standard, in §3.403:

The standard defines the term and then tells you the system doesn’t enforce it.

Nothing in the filesystem records “this is text.” There’s no flag on the inode, no attribute, nothing in the directory entry. The .txt extension is a hint to humans and to Windows. “Text file” is not a property a file has. It’s a claim the reader makes about the bytes, and every tool makes it slightly differently.

The Bytes Don’t Know What They Mean

A file stores bytes. Turning bytes into characters requires an encoding, and the encoding is not in the file.

Here are the same five characters, Héllo, in several encodings:

ascii         FAILS: ordinal not in range(128)
latin-1        5 bytes  48 e9 6c 6c 6f
cp1252         5 bytes  48 e9 6c 6c 6f
utf-8          6 bytes  48 c3 a9 6c 6c 6f
utf-16        12 bytes  ff fe 48 00 e9 00 6c 00 6c 00 6f 00
utf-16-be     10 bytes  00 48 00 e9 00 6c 00 6c 00 6f
utf-32        24 bytes  ff fe 00 00 48 00 00 00 e9 00 00 00 ...

Five characters. Anywhere from 5 to 24 bytes. Nothing in any of those files says which one you’re looking at.

When you open a file in your editor and it looks right, that’s your editor guessing correctly. When you get caf√© instead of café, that’s your editor guessing wrong. The file never changed.

The Fight Over the Eighth Bit

ASCII was standardized as a 7-bit code: 128 values, 0x00 through 0x7F. Thirty-three control codes, ninety-five printable characters, and that was the whole world if the world spoke English.

Bytes have eight bits though, so there were another 128 values sitting there unused. Everyone grabbed them, and everyone grabbed them differently.

ISO 8859-1 (Latin-1) claimed 0xA0–0xFF for Western European letters and reserved 0x80–0x9F for a second set of control codes nobody used. Microsoft looked at those 32 wasted slots and put printable punctuation there instead, creating Windows-1252. That’s where the curly quotes and the em dash live:

CP1252 text  : It's "fine" — really
CP1252 bytes : 49 74 27 73 20 93 66 69 6e 65 94 20 97 20 72 65 61 6c 6c 79
UTF-8 bytes  : 49 74 27 73 20 e2 80 9c 66 69 6e 65 e2 80 9d 20 e2 80 94 ...

Byte 0x93 is a left curly quote in CP1252 and a control character in strict Latin-1. This is why pasting from Word into a system expecting Latin-1 produces garbage: the bytes are legal, they just mean nothing there.

Mojibake, Japanese for “character transformation,” is exactly this, and it’s completely deterministic:

original text  : café
as UTF-8 bytes : 63 61 66 c3 a9
read as CP1252 : cafÃ©

c3 a9 is one character in UTF-8 and two characters in CP1252. Both readings are valid. Only one is what you meant.

It could have been worse. IBM’s EBCDIC, still running on mainframes, isn’t an ASCII superset at all:

'A'  ASCII 0x41   EBCDIC 0xc1
'a'  ASCII 0x61   EBCDIC 0x81
' '  ASCII 0x20   EBCDIC 0x40

And the letters aren’t even contiguous. I is 0xc9, J is 0xd1, with a gap in between. Sorting strings by byte value, which works fine in ASCII, silently produces wrong output in EBCDIC.

Why UTF-8 Won

UTF-8 encodes a character in one to four bytes. ASCII characters keep their single-byte values, so every ASCII file is already a valid UTF-8 file. That backward compatibility gets most of the credit, but the more interesting property is the bit pattern:

'A'  U+0041   1 byte   41           01000001
'é'  U+00E9   2 bytes  c3 a9        11000011 10101001
'€'  U+20AC   3 bytes  e2 82 ac     11100010 10000010 10101100
'🙂' U+1F642  4 bytes  f0 9f 99 82  11110000 10011111 10011001 10000010

Look at the leading bits. A single-byte character starts with 0. A multi-byte character starts with 110, 1110, or 11110, where the number of leading 1s is the total byte count. Every continuation byte starts with 10, and nothing else does.

That makes UTF-8 self-synchronizing. Drop into the middle of a file at a random offset and you can tell immediately whether you’re mid-character, and walk backwards a byte or two to find the boundary. You do not need to have read the file from the beginning.

Compare that to UTF-16, where you must know the byte order and must have tracked whether you’re on an even or odd boundary. UTF-8 made encoding a local property instead of a global one, and that’s why it took over.

The BOM

Multi-byte encodings have a byte order problem: is 00 48 the character U+0048 or U+4800? The Byte Order Mark solves it by putting U+FEFF at the start of the file, so a reader can look at the first two bytes and work out the endianness.

utf-16      ff fe 68 69 ...   (little-endian)
utf-16-le   68 00 69 00       (no BOM, you'd better know)
utf-8-sig   ef bb bf 68 69    (UTF-8 "BOM")
utf-8       68 69             (no BOM)

UTF-8 has no byte order to mark, because its unit is one byte. The UTF-8 BOM is not a byte order mark at all; it’s a three-byte flag saying “this is UTF-8,” and the Unicode Consortium neither requires nor recommends it.

It also actively breaks things. The kernel identifies a script by looking for 0x23 0x21, the characters #!, at offset zero:

no BOM  : 23 21 2f 62 69 6e 2f 73 68 0a  ->  #!/bin/sh
with BOM: ef bb bf 23 21 2f 62 69 6e 2f  ->  not a script

Same for JSON parsers, CSV importers, and anything else that expects a specific first byte. If you have ever seen a shell script fail with a cryptic error on a line that looks correct, this is a candidate.

Lines Are a Convention Too

There is no line structure in a text file. There’s a byte that tools agree means “line break,” and even that isn’t agreed on.

The split is a hardware inheritance. A teletype needed two separate mechanical actions to start a new line: carriage return (0x0D) moved the print head back to the left margin, and line feed (0x0A) advanced the paper by one row. Two actions, two control codes.

Then everyone picked differently. Unix chose LF alone. MS-DOS, and Windows after it, kept both as CRLF. Classic Mac OS used CR alone. Those choices are still with us thirty years later, and they’re the reason .gitattributes exists.

And then there’s the trailing newline, which people argue about without realizing the standard already answered it. POSIX §3.206:

The newline is part of the line, not a separator between lines. A file whose last byte isn’t a newline doesn’t have a final line. POSIX §3.195 has a name for what it has instead: an incomplete line.

That definition has teeth:

$ wc -l lf.txt nofinal.txt
       2 lf.txt
       1 nofinal.txt

Both files contain the text one and two. The first ends with a newline, the second doesn’t. wc -l counts newline bytes, so the second file reports one line despite visibly having two.

This is also what git’s \ No newline at end of file marker means. It isn’t a style complaint. Git is telling you the last line is incomplete by the POSIX definition, which matters because otherwise appending a line would silently modify the existing last line rather than adding a new one.

How Tools Guess

Since nothing declares itself, every tool that needs to know applies a heuristic. The dominant one is: does it contain a NUL byte?

That test exists because C strings are NUL-terminated, so a NUL in the middle of what claims to be text means something is off. It’s a good heuristic. It’s also wrong in two ways worth knowing about.

Git’s version is buffer_is_binary() in xdiff-interface.c, and it doesn’t scan the whole file. It caps at a constant:

#define FIRST_FEW_BYTES 8000

So the check is “is there a NUL in the first 8000 bytes.” A file with clean text for 10KB and a NUL after that is text as far as git is concerned. The cutoff is a performance tradeoff, and it means binary-ness is decided by a sample, not a proof.

The second problem is bigger.

plain ascii          NUL present: False  -> text
utf-8 with emoji     NUL present: False  -> text
has a NUL byte       NUL present: True   -> BINARY
utf-16 text          NUL present: True   -> BINARY

UTF-16 encodes ASCII characters as the character byte plus a NUL. Any UTF-16 file that’s mostly English is roughly half NUL bytes. So git does this to a perfectly valid text file:

$ git diff --cached --stat
 lf.txt      |   2 ++
 utf16.txt   | Bin 0 -> 24 bytes

Bin. Git will not diff it, will not merge it, and will not show it in review. The heuristic isn’t detecting text, it’s detecting C-string-safety, and those aren’t the same question.

file(1) is more thorough, and it shows how much the BOM is doing:

$ file utf16.txt utf16_bom.txt lf.txt
utf16.txt:     data
utf16_bom.txt: Unicode text, UTF-16, little-endian text
lf.txt:        ASCII text

Identical text content in the first two files. The only difference is two leading bytes. Without them file gives up and calls it data; with them it identifies the encoding exactly. For a format with no header, a BOM is the closest thing to one that exists.

Under the hood file is doing real work rather than one heuristic. src/encoding.c carries a 256-entry table classifying every byte value as never-valid-in-text, ASCII, ISO-8859, or extended ASCII, plus a dedicated UTF-8 state machine that rejects invalid sequences. It then tries candidate encodings in order: ASCII, UTF-7, UTF-8 with BOM, UTF-8, UTF-32, UTF-16, Latin-1, extended ASCII, and finally EBCDIC. That ordering is a nice fossil record of which encodings are still worth guessing first.

Why This Matters

Nearly every format developers work in daily is a convention layered on this substrate. Source code, JSON, YAML, TOML, CSV, Markdown, config files, logs. All of them inherit these problems, and none of them can fully escape them, because the layer underneath has no way to describe itself.

That’s the tradeoff. A format with no header can’t tell you anything about itself, which is exactly why it has outlived every format that could. PNG will be readable as long as someone maintains a PNG decoder. A text file is readable as long as someone remembers what bytes are.

Next in the series: Markdown, which is a text file plus a set of conventions that nobody fully agrees on.

Sources

POSIX.1-2017 Base Definitions, Chapter 3 — §3.206 Line, §3.195 Incomplete Line, §3.403 Text File
RFC 3629 — the UTF-8 specification and its byte patterns
RFC 2046 §4.1 — the text/plain media type
Unicode FAQ on UTF-8, UTF-16, and the BOM — the Consortium’s own guidance on why not to use a UTF-8 BOM
buffer_is_binary() in git’s xdiff-interface.c — the NUL check and the FIRST_FEW_BYTES cutoff
src/encoding.c in the file project — the text-character table and encoding-guessing order behind file(1)

How PNG Actually Stores Your Pixels

Fri, 14 Aug 2026 10:00:00 -0500

I’m starting a series on file formats. Not “here are the ten image formats you should know,” but the actual bytes: what’s in the file, in what order, and why someone decided it should be that way.

Starting with PNG, because it’s the format most developers touch every day and almost nobody has looked inside.

I am likely to cover a few things that other explainer documents have covered, such as chunk structure and chunk types. However, I’d like to dig into some details that are not often mentioned, such as where your pixels went.

A Format Born From a Patent Fight

PNG exists because of a licensing ambush. On 28 December 1994, right in the middle of the holidays, Unisys announced an agreement to start collecting royalties from authors of GIF-supporting software, on the strength of its patent on the LZW compression algorithm that GIF used.

The response was fast. A draft for a replacement format was posted to comp.graphics on 4 January 1995, one week later. It was originally called PBF, for Portable Bitmap Format, and got renamed to PNG two days after that. The format shipped as a W3C Recommendation in October 1996.

Two things about that origin still show in the bytes. The format is aggressively defensive, because it was designed by people who expected files to be mangled in transit. And it is aggressively extensible, because they had just watched a format become unusable for reasons that had nothing to do with its technical design.

Eight Bytes of Paranoia

Every PNG starts with the same eight bytes:

Hexadecimal:  89  50  4E  47  0D  0A  1A  0A
ASCII/Ctrl: \x89  P   N   G  \r  \n \x1A \n

P N G in the middle is obvious. The other five bytes are a booby trap for 1995-era file transfer, and each one catches a specific failure:

0x89 has the high bit set. Some 7-bit transfer paths stripped bit 7 from every byte. If that happened, this byte arrives as 0x09 and the file is detectably wrong on byte one.
0x0D 0x0A is a DOS line ending. A text-mode FTP transfer that “helpfully” converts CRLF to LF mangles it.
0x1A is Ctrl-Z, the MS-DOS end-of-file marker. If you TYPE a PNG at a DOS prompt, output stops here instead of spraying binary at your terminal and leaving it in a weird state.
0x0A is a bare LF, catching the opposite conversion: LF silently expanded to CRLF.

Who would have thought that so many bits were used just to account for line endings in different operating systems? I suppose it’s good to plan ahead when designing a file format.

Everything Is a Chunk

After the signature, a PNG is a flat sequence of chunks. No central directory, no offset table. You read them in order.

Every chunk has the same four-field shape:

Field	Size	Notes
Length	4 bytes	Big-endian, counts only the data field
Chunk Type	4 bytes	Four ASCII letters
Chunk Data	Length bytes	Can be zero-length
CRC-32	4 bytes	Computed over type and data, not over length

Two details worth keeping. The length field is 32 bits but the spec caps values at 2³¹−1, so the high bit is always clear. And the CRC covers the type plus the data but skips the length, which means a corrupted length field is not detected by the chunk’s own checksum.

The chunk type is where PNG does something clever. Those four letters are ASCII, and bit 5 of an ASCII letter is what distinguishes uppercase from lowercase (A is 0x41, a is 0x61). PNG uses that bit in each of the four positions as a flag:

Position	Uppercase means	Lowercase means
1st	Critical: decoder must understand it	Ancillary: safe to ignore
2nd	Public, registered in the spec	Private, vendor-specific
3rd	Reserved, must be uppercase today	(reserved for future use)
4th	Unsafe to copy if pixels changed	Safe to copy blindly

So a decoder that has never heard of tEXt can tell from the lowercase t that skipping it is fine. A decoder hitting IDAT sees the uppercase I and knows it cannot skip it. The capability negotiation is encoded in the name itself, which means you can add chunk types decades later without breaking old readers. This is why APNG could bolt animation onto PNG without a version bump.

Four chunk types are critical: IHDR (header, always first), PLTE (palette), IDAT (the pixels), and IEND (a zero-length terminator).

IHDR Is the Decode Key

IHDR is exactly 13 bytes and it comes first because nothing else can be interpreted without it:

Width (4 bytes) and Height (4 bytes), big-endian
Bit depth (1 byte): bits per sample, one of 1, 2, 4, 8, 16
Color type (1 byte): what a pixel is made of
Compression method (1 byte): always 0
Filter method (1 byte): always 0
Interlace method (1 byte): 0 for none, 1 for Adam7

Bit depth and color type together determine everything about the pixel layout, and only certain combinations are legal:

Color type	Name	Samples per pixel	Legal bit depths
0	Greyscale	1	1, 2, 4, 8, 16
2	Truecolor (RGB)	3	8, 16
3	Indexed	1 (a palette index)	1, 2, 4, 8
4	Greyscale + alpha	2	8, 16
6	Truecolor + alpha (RGBA)	4	8, 16

Note the gaps. You cannot have 16-bit indexed color, because a palette holds at most 256 entries and 8 bits already addresses all of them. You cannot have 1-bit RGB, because a “1-bit red sample” isn’t a useful thing. The table isn’t arbitrary; each missing cell is a combination that would be incoherent.

Also note that bit depth is per sample, not per pixel. A bit depth of 16 with color type 6 means 16 bits each for R, G, B, and A: 64 bits per pixel. That’s the “64-bit RGBA” you see in PNG marketing.

Where the Pixels Actually Live

Uncompress all the IDAT data and concatenate it, and you get a byte stream. That stream is not a grid. It’s a sequence of scanlines, one per image row, top to bottom. And each scanline is:

[1 filter type byte][packed sample data for the whole row]

That leading byte is not pixel data. It’s a number from 0 to 4 saying which filter was applied to this row.

The sample data is packed with no padding between pixels and no separators. Samples appear in a fixed order within each pixel:

Greyscale: grey
Truecolor: red, green, blue
Indexed: palette index
Greyscale + alpha: grey, alpha
Truecolor + alpha: red, green, blue, alpha

Here is an example PNG, filter type 0 (no filtering) on both rows:

scanline 0: 00 ff 00 00 00 ff 00 00 00 ff ff ff 00
            ^^ filter byte
            ^^^^^^^^ red pixel (ff,00,00)
                     ^^^^^^^^ green pixel (00,ff,00)

scanline 1: 00 00 00 00 80 80 80 ff ff ff ff 00 ff
            ^^ filter byte
               ^^^^^^^^ black    ^^^^^^^^ white

Twelve bytes of pixel data per row (4 pixels × 3 samples), each prefixed by one filter byte, for 26 bytes of raw stream. The complete file, signature and all four chunks included, is 83 bytes.

The whole model at 8-bit depth: walk the row, emit samples in order, move on. No alignment, no padding, no per-pixel headers.

Bit depths of 1, 2, and 4 only apply to greyscale and indexed images. Multiple pixels get packed into a single byte.

Leftmost pixel goes in the high bits. So for a 12-pixel-wide 1-bit greyscale image:

pixels       : 1 1 0 1 0 0 0 1  1 0 1 1
packed bytes : 0xd1 0xb0
               11010001 10110000
                              ^^^^ unused

Twelve pixels need 12 bits, which rounds up to 2 bytes, leaving 4 bits spare at the end. The spec’s language on those leftover bits:

Scanlines always start on a byte boundary. Row 2 never continues in the leftover bits of row 1’s last byte.

At bit depth 16, each sample is two bytes, most significant byte first. The spec calls it network byte order. On x86 and ARM, which are little-endian, that means every 16-bit sample needs a byte swap on read and on write.

The Filter Byte Is the Whole Trick

Now back to that leading byte on every scanline.

PNG uses DEFLATE, the same algorithm as gzip and zip. If you just DEFLATE’d raw pixels, PNG would compress about as well as gzipping a bitmap, which is to say barely at all. Photographs and gradients don’t repeat exact byte sequences, and LZ77 needs exact repeats.

So before compressing, PNG transforms each scanline into differences from its neighbors. Five filters are available, chosen per scanline:

Type	Name	Transform
0	None	store the byte as-is
1	Sub	subtract the byte from the pixel to the left
2	Up	subtract the byte from the pixel above
3	Average	subtract the average of left and above
4	Paeth	subtract whichever of left/above/upper-left is the best predictor

All arithmetic is mod 256, which is what makes it reversible without storing a sign. And “the pixel to the left” means the byte at the same position in the previous pixel, so for RGB the red sample is compared against the previous red sample, not against the previous blue.

Take a 16-pixel greyscale gradient stepping by 10:

raw scanline     : 00 0a 14 1e 28 32 3c 46 50 5a 64 6e 78 82 8c 96
after Sub filter : 00 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a 0a

Sixteen distinct byte values become two. The image is unchanged and the transform is exactly reversible, but LZ77 now sees a run it can encode in almost nothing. On this toy row, DEFLATE produces 28 bytes for the raw version and 15 for the filtered one.

Sixteen bytes is far too small for DEFLATE to stretch its legs, so don’t read that ratio as typical. The point is the entropy collapse: filtering doesn’t compress anything, it rearranges the data so the compressor has something to find.

That per-scanline choice is also why two encoders produce different-sized files from identical pixels. libpng, ImageMagick, oxipng, and zopflipng all ship different filter-selection heuristics. Same spec, same decoded output, different bytes on disk. Most PNG optimizers are search algorithms over filter choices, not better compressors.

The Compression Pipeline, End to End

Putting it together:

raw pixels
  -> pack into scanlines (samples in order, sub-byte packing if needed)
  -> prepend a filter byte per scanline, apply the filter
  -> DEFLATE the whole concatenated stream (LZ77 + Huffman)
  -> wrap in a zlib container (RFC 1950)
  -> split across one or more IDAT chunks

A few consequences fall out of that ordering:

The zlib stream spans chunks. IDAT boundaries are arbitrary. A decoder must concatenate every IDAT payload and then decompress; decompressing them individually fails. Encoders split them for streaming, not for structure.

The Adler-32 checksum in the zlib wrapper covers filtered bytes, not your original pixels. It validates decompression, not image fidelity. The per-chunk CRC-32 is what protects against transmission corruption.

Compression is global across the image. LZ77’s 32KB sliding window means row 400 can match against row 380 if they’re similar. This is why a 64×64 solid color block compresses to 136 bytes while a 64×64 gradient of the same dimensions takes 10,362 bytes, against 12,288 bytes raw. Uniformity compresses; novelty doesn’t.

And a practical one: for that solid-color block, encoding as indexed color with a one-entry palette produces a 99-byte file instead of 136, because each pixel is one index byte instead of three samples. If your image has few colors, color type 3 usually beats truecolor even after DEFLATE gets its turn.

Interlacing, Briefly

If the interlace byte in IHDR is 1, the image uses Adam7: the pixels are transmitted in seven passes over an 8×8 grid, coarse to fine, so a partially-downloaded image renders as a low-resolution preview that sharpens.

Two things to know. Each pass is filtered and encoded as an independent sub-image with its own scanlines and filter bytes, so a decoder can’t treat the stream as one grid. And Adam7 typically makes files larger, because breaking the image into seven sparse sub-images destroys exactly the local coherence that filtering and LZ77 depend on. It was a good trade on a 28.8k modem. On any modern connection it costs size and complexity for a progressive render nobody waits around to see.

What This Buys You

The design decisions hold up well for a 1996 format:

Unknown chunks are safe by construction, so the format extended to EXIF metadata, ICC profiles, and animation without ever breaking old decoders.
Every chunk is individually checksummed, so corruption is localized and detectable rather than silently rendering garbage.
Filtering is a preprocessing step, not a compression format, which means encoders can get better forever without touching the spec. A file written by zopflipng today decodes fine in a 1997 reader.

That extensibility is not just historical. PNG got a Third Edition as a W3C Recommendation on 24 June 2025, which finally standardized APNG, added an eXIf chunk for camera metadata, and brought in HDR through three new chunks (cICP, mDCV, cLLI). Thirty years on, the container still had room.

Where it shows its age is DEFLATE, which is a 1990s compressor. Lossless WebP does beat it: Google’s own study puts WebP lossless at 23% smaller than PNGs already optimized with ZopfliPNG, and 42% smaller than default libpng output. Worth noting the baseline matters enormously there, and Google’s WebP FAQ quotes a different figure (26%) than the study it links to.

Lossless AVIF is a murkier story than the marketing suggests. AOMedia publishes no general lossless-AVIF-versus-PNG number at all; its quantified claims (50% versus JPEG, 30% versus WebP) are all about lossy encoding. The only primary figure available is 10% versus a 16-bit PNG for a single demo image using a new v1.2.0 feature. Independent testing regularly finds lossless AVIF producing larger files than PNG for flat synthetic images like icons, UI, and charts. If you’re picking a format for screenshots and diagrams, test on your own images rather than trusting a general ranking.

Next in the series: the opposite of all this. A text file, which announces nothing about itself at all.

Sources

W3C PNG Specification, Third Edition — the current standard; §7.2 covers scanlines and sample packing, §9 covers filtering
RFC 2083 — the original 1997 IETF PNG specification
RFC 1950 (zlib) and RFC 1951 (DEFLATE) — the compression layer
libpng PNG history — the Unisys announcement, the PBF name, and the January 1995 timeline
WebP Lossless and Alpha Study — Google’s 23%/42% lossless figures and their baselines
AOMedia on AVIF v1.2.0 — the 10% lossless figure, and its narrow scope

Deploying Hermes Agent With Ansible Without Creating a Snowflake

Thu, 13 Aug 2026 10:00:00 -0500

I have a home server with Plenty of RAM and no useful GPU, so running a local model was never the interesting part of deploying Hermes Agent. The interesting part was making the agent setup repeatable.

I could have pasted a docker run command over SSH and called it finished. It would have worked. But “it works” and “I can rebuild this server six months from now” are two very different things.

So I built an Ansible role around the official Hermes container, authenticated with a ChatGPT subscription through the openai-codex provider, and left the one interactive step, OAuth, outside Ansible. That split is the whole idea. Ansible owns the infrastructure. Hermes owns its refresh token. I own the browser login.

Start With the Deployment Contract

Before writing a single task, decide what the role is promising. Mine had five rules:

Pin the image by version and digest. A mutable latest tag is not a deployment plan.
Persist /opt/data. Hermes keeps configuration, sessions, skills, memories, and authentication state there.
Publish the API on 127.0.0.1 only. A private Docker network handles future service-to-service access.
Bound the process. 4 GiB of RAM, 2 CPUs, 256 PIDs, bounded logs, and agent loop limits.
Keep OAuth manual. Ansible creates the service, then the operator completes the device-code login over SSH.

The role layout is conventional: defaults/main.yml, handlers/main.yml, tasks/main.yml, and templates for the Compose file, the Hermes config, the env file, and a host wrapper script.

Put every value you expect to tune in defaults/main.yml:

hermes_agent_home: /home/youruser/Web/hermes-agent
hermes_agent_data_dir: "{{ hermes_agent_home }}/data"

hermes_agent_image: >-
  nousresearch/hermes-agent:v2026.8.3@sha256:<tag>
hermes_agent_api_port: 8642
hermes_agent_publish_host: "127.0.0.1"

hermes_agent_model_provider: openai-codex
hermes_agent_model_name: gpt-5.6-terra
hermes_agent_memory_limit: 4g
hermes_agent_cpu_limit: "2.0"
hermes_agent_pids_limit: 256

Use a release and digest you have reviewed. The version above is what I deployed, not a promise that it is still the right one when you read this.

The Compose template turns those defaults into an enforceable boundary:

services:
  hermes:
    image: {{ hermes_agent_image }}
    command: ["gateway", "run"]
    restart: unless-stopped
    env_file:
      - ./hermes.env
    volumes:
      - {{ hermes_agent_data_dir }}:/opt/data
    ports:
      - "127.0.0.1:8642:8642"
    mem_limit: 4g
    cpus: "2.0"
    pids_limit: 256
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8642/health"]
      interval: 10s
      retries: 12
    logging:
      options:
        max-size: "10m"
        max-file: "5"

The loopback bind matters. The official Hermes Docker docs recommend authenticated access for exposed services, and specifically call out SSH tunnels or private networking as the safer way to reach a loopback-bound dashboard. I don’t need the API listening on every interface just because Docker makes that easy.

Make Ansible Own Setup and Proof

The role should do more than render YAML. It should reject unsafe input before it mutates anything, create persistent directories with deliberate ownership, reconcile Compose, and prove the service came back.

- name: Validate Hermes configuration
  ansible.builtin.assert:
    that:
      - hermes_agent_api_key | length >= 32
      - hermes_agent_publish_host in ['127.0.0.1', '::1']
      - hermes_agent_model_provider in ['openai-codex', 'openai-api', 'openrouter']
  no_log: true

- name: Render protected environment
  ansible.builtin.template:
    src: hermes.env.j2
    dest: "{{ hermes_agent_home }}/hermes.env"
    mode: "0600"
  no_log: true

- name: Reconcile Hermes Compose project
  community.docker.docker_compose_v2:
    project_src: "{{ hermes_agent_home }}"
    state: present
    pull: missing

Notice the two uses of no_log. An encrypted variable is protected at rest, but Ansible will happily reveal the decrypted value in task output or --diff. Secret-bearing template and validation tasks should not print their inputs.

I encrypted only the API bearer key, not the whole variables file:

ansible-vault encrypt_string \
  --vault-password-file vault_password_file \
  --stdin-name vault_hermes_agent_api_key

Type the value, press Ctrl-D, paste the resulting !vault block into your group vars, and let a no_log assertion check its length. You don’t need to print the plaintext back into your terminal to prove Ansible can decrypt it.

I also wanted keyless web search, so the role installs a pinned DDGS package into persistent storage with uv, not pip:

- name: Install pinned DDGS
  community.docker.docker_container_exec:
    container: hermes-agent
    user: hermes
    argv:
      - /usr/local/bin/uv
      - pip
      - install
      - --python
      - /opt/hermes/.venv/bin/python
      - --target
      - /opt/data/lazy-packages
      - --reinstall
      - ddgs==9.14.4

The follow-up check imports DDGS, verifies the version, and confirms /opt/data/lazy-packages is on sys.path. Checking for a metadata directory is not enough.

One gotcha from the release I deployed: do not render HERMES_YOLO_MODE=0. Its mere presence still triggered the YOLO banner. If you want manual approvals, omit the variable entirely and set the approval mode in the Hermes config instead.

OAuth stays manual, because browser authentication is an operator action, not configuration management:

ssh user@homelab
docker exec -it hermes-agent hermes auth add openai-codex --no-browser
docker exec hermes-agent hermes auth status openai-codex

The refreshable credential lands under the persisted /opt/data directory. Do not copy your laptop’s ~/.codex directory into the container just to skip one login.

Make Daily Use Boring Too

The last piece was a host command. I wanted to SSH into the server, type hermes, and resume the last session without remembering a Docker incantation.

#!/bin/sh
set -eu

if [ "$#" -eq 0 ]; then
  set -- --continue
fi

if [ -t 0 ] && [ -t 1 ]; then
  exec docker exec -it -w /opt/data/workspace hermes-agent hermes "$@"
fi

exec docker exec -i -w /opt/data/workspace hermes-agent hermes "$@"

Install that template as /usr/local/bin/hermes with mode 0755. Arguments pass through, so hermes --help and hermes auth status openai-codex work from the host too.

Then run the play three times: once with --check --diff to preview, once to deploy, and once more to confirm it reports changed=0. Idempotence you haven’t observed is idempotence you’re guessing at. Finish by checking the container is healthy and the port is where you think it is:

docker inspect --format '{{.State.Health.Status}}' hermes-agent
docker port hermes-agent

That is the difference between a container I happen to have running and a service I know how to rebuild. The manual OAuth step isn’t a failure of automation, it’s a clean boundary around a human credential flow.

Exactly what I want from Ansible.

Sources & References

Hermes Agent Docker guide — official container deployment docs
Hermes Agent provider documentation — including openai-codex
Ansible Vault: encrypting individual variables
community.docker.docker_compose_v2

pnpm 11 Made the Safe Thing the Default

Wed, 12 Aug 2026 10:00:00 -0500

Protecting against supply chain attacks requires vigilance. You have to audit your dependencies. You have to pin your versions. You have to review your install scripts. All of these are great things to do, but they require sustained effort.

pnpm 11 took the obvious thing and made it the default. They changed the waiting period. minimumReleaseAge defines the minimum number of minutes that must pass after a version is published before pnpm will install it.

Before version 11 the default was 0. In version 11 the default is 1440 minutes, which is 24 hours, and it applies to everything.

Most malicious packages get discovered and pulled from the registry within minutes, or at most an hour. So what a day of patience buys you is that you’ve eliminated the potential for dependencies sneaking in that haven’t been fully vetted.

npm already followed suit, and so did everyone else. This is now table stakes across the ecosystem:

pnpm got there first with minimumReleaseAge, measured in minutes, back in 10.16 in September 2025.
Yarn shipped npmMinimalAgeGate, also minutes, in 4.10.0 that same month.
Bun added minimumReleaseAge in 1.3 in October 2025, measured in seconds, plus a minimumReleaseAgeExcludes list for packages you trust.
npm landed min-release-age in 11.10.0 in February 2026, measured in days.

I guess they couldn’t agree on the unit of time for their release age setting.

As far as I know, in all of them besides pnpm, the cooldown is opt-in. It’s not the default. So you have to know the setting exists and you have to go turn it on.

Here are some other things that changed in pnpm 11:

allowBuilds replaces onlyBuiltDependencies, which was removed in v11. It’s a map of which packages may run build scripts. Anything not listed is disallowed and treated as unreviewed.
strictDepBuilds defaults to true. Installation exits with a non-zero code if any dependency has unreviewed build scripts, so this fails your CI rather than printing a warning nobody reads.
verifyDepsBeforeRun defaults to install. Before pnpm run or pnpm exec, it checks whether your dependency state matches the lockfile. Other options are warn, error, prompt, and false.
dangerouslyAllowAllBuilds defaults to false, and the name is doing exactly the work it should. Setting it true lets every dependency, transitive ones included, run install scripts now and in the future.

The clear pattern here is that an automated or unintentional action should be blocked, not permitted with a warning.

Sure, there is somewhat of a cost here. The delay means you can’t immediately install a new version that was just published unless you flip the flag. I can see the allowBuilds migration being somewhat of a hassle, because the first time you install after upgrading you’re going to get a list of packages that want to run build scripts. It’s easy to be lazy and approve all of them without thinking.

With the tools we have available to us these days, we can ask an agent to review the build scripts. This is the right thing to do. Find a way to pin the dependency to fix transitive version issues. The inner engineer in all of us needs to understand why build scripts are dangerous, and what to be careful of, so that you can ask your subagent to go and see if that’s a problem, or if that problem has been fixed with the new version. It’s up to the human in the loop to ensure that the agents are doing their due diligence.

When building software, we should also look for other paths to optimize, and make the lazy path the safest one, because chances are that’s going to become the default.

Onboarding a Second Engineer to an Agent-Heavy Repo

Tue, 11 Aug 2026 10:00:00 -0500

How do you onboard a new engineer to a project when most of what they need to know is not checked into the code at all? It’s the agent configuration. It’s the rules about commands, the conventions the agent applies automatically because it’s stored in your memory. That isn’t shared.

I think we’re still figuring out the answers to this. I’m not sure that we should be sharing the context file. I don’t think we should be sharing subagents. I could see skills being a shared resource.

I think it’s more important that you don’t share everything. The things that should be shared, share cleanly. Set it up like you would set up linting rules on your pipeline.

Understanding how the other person intends to be productive will impact what items need to be shared. Having access to a good memory system is way more important than checking in your context file.

When a new person joins, of course, give them access to the code. Of course, give them access to things that you’ve agreed upon should be shared. But you should also talk with them and understand what their expectations are of a real workflow from end to end. Do they have the tools that they need in order to build reliable and repeatable workflows that will get results?

Access to the tools that they need is more important than sharing every single tiny detail.

People responsible for making changes to a system can learn a lot by just doing. You need to understand the tools before you can understand the system.

Boring Is a Feature

Mon, 10 Aug 2026 10:00:00 -0500

What does boring look like in the age of AI? And I’m not talking about uninteresting. I’m talking about highly maintainable.

JavaScript?

I mean I guess models are good at it. Everybody knows it. It runs everywhere. The biggest problem with JavaScript is that TypeScript is better.

Certainly it’s better than picking a novelty language that you haven’t built anything with before. Does anybody on the team actually know Haskell? And how long ago did they know Haskell? You need to evaluate the cost of adopting it just as you would evaluate how long it would take to learn it and train the team on it.

The upfront cost can be easier to measure. But the recurring ones are much harder to predict. What happens when a maintainer moves on from a package that everyone uses, and the speed it takes to find a new maintainer is not as fast as you need it to be?

Boring tools are ones where the recurring cost of maintenance is as close to zero as you can get it. If you come back to it in eight months, it should work the way you remember it. This is a fairy tale that we tell ourselves, that nothing ever changes and we can control that change.

Is boring even possible in the age of AI, when it feels like everyone has their own particle beam cannon that they can point at your codebase?

I think it’s worth talking about what I mean by boring, because it can be used as a synonym for old, but that’s not what I mean. Boring means predictability.

Take all your npm packages. Can you answer these questions about all of them? Probably not.

How often do the release notes contain the word “breaking”? Skim a year of them. This is the single best signal available and it takes ten minutes.
How many people can merge? One is a risk regardless of how good that one person is. People change jobs, burn out, and lose interest.
What happens to old versions? A project that supports the previous major for a while is telling you something about how it thinks about your time.
Can you read the source? Not all of it. Enough to fix something yourself when you’re blocked and nobody’s answering.

Learn the new tool. Experiment. Try new things. Stay passionate about software. Just because you can use the new thing doesn’t mean you should.

Don’t always pick the boring option, just like you don’t always pick the new option. It takes wisdom to know what the right answer is.

You have to understand your failure modes, and when it’s an appropriate time to take a risk, and the scale of the risk.

Pick boring for the parts you don’t want to think about. Save the interesting decisions for the places where being interesting is the point.

The Best Automation Has a Manual Escape Hatch

Sun, 09 Aug 2026 10:00:00 -0500

Automation earns trust by being easy to override, not by being impossible to question.

That sounds backwards. The pitch for automating something is usually that it removes the human, and a system you keep reaching into feels like a system that didn’t finish the job. But the automation you actually trust, over years, is the one you know you can stop.

Most automation that you set up is enforcing some sort of policy, and that’s right most of the time, but not always.

The mistake isn’t automating a default way of working. It’s building a system where the default is ingrained so deeply that there’s no way out of it.

The automation must be flexible. You must be able to adapt the automation as the requirements change.

Do you have contingency plans on what to do if the automation fails?

Now I’m not talking about how to get around the automation, or always forcing an outcome that disables the automation. Instead, I’m talking about what a real escape hatch looks like.

It’s one operation. You run a command. You don’t perform a sequence of five steps where forgetting the third leaves things inconsistent.

It maintains the invariants. This is the big one. When I override a post’s date, the file and the database both get updated. If the override only touched one of them, I’d have created a split-brain problem in the name of fixing a scheduling problem.

It’s discoverable. It shows up in the help output next to everything else. An escape hatch nobody knows about is not a feature, it’s trivia.

It’s supported, not tolerated. It has tests. It survives refactors. Nobody has to feel clever for using it.

If your answer to “what if the automation is wrong” is “go around it manually,” you don’t have a hatch. You have a hazard with a tradition attached.

If you design the escape hatch first, it forces a question that’s worth thinking about. At least what happens when the automation is wrong. What are your plans to do something about it?

Log When the Hatch Gets Used

Don’t forget about the log. It’s not one that you should skip over. You should be logging when your escape hatch gets used, even if it only happens once a quarter.

You probably don’t need to update your policy every time. But your escape hatch log is a good indication of when you might consider updating the policy.

Building an escape hatch changes the risk. The worst case is not that the tool did something irreversible, but rather that the tool did something I fixed in one command.

So build the hatch. Make it one command, make it maintain your invariants, put it in the help text, and count how often it gets pulled.

The automation you trust isn’t the one that’s always right. It’s the one you know you can overrule.

Cost and Latency Belong in the Score

Sat, 08 Aug 2026 10:00:00 -0500

Congratulations, the best model available passed your eval. That’s not the question you should be answering. What other models could have achieved equivalent results? How much did it cost to run those other models? How long did it take for those other models to achieve equivalent results? All of these are the questions that you should be asking yourself.

If the only thing you’re measuring is if the output is correct or not, then we already know the ranking. The bigger models are going to score higher on the majority of tasks. You don’t need an eval harness to answer that question. Quality scoring is only part of the answer that you actually care about. On every eval run, you should be recording at least three things per task.

Quality. Whatever your pass or fail criteria are. This is the part you already have.
Cost. What the run consumed. Tokens in, tokens out, and the price attached to them.
Latency. Wall clock, start to finish.

These aren’t particularly hard to capture. Cost and latency are usually right there in the response and you’re probably throwing them away. Latency is a hill to climb in its own right, given that it can vary depending on the reliability of the models. Latency at different times of day also matters more than at others. If the human operator is asleep, then latency matters less.

Once you start recording these three dimensions, the promise is you’ll be able to more accurately answer the question: which model is best?

I mean that’s the whole point of all this, right?

What you’re really building is a system, or an attempt at a system, for predicting the output of a non-deterministic system.

Good luck.

A global spending cap for the month doesn’t answer the question: are you using the wrong model for particular tasks?

Task routing is complicated. I can see it being a complex problem to try to solve.

You’re going to find that cheap models win more often than you’d think. There’s a whole bunch of types of work where you don’t need a frontier model, where you’re just using the wrong tool for the task at hand. Think about reformatting, or extracting data from structured text, or classifying something into a bucket, or summarizing a document, or renaming things. All of these are narrow tasks that don’t require a lot of reasoning between the input and the output.

One thing to try is to do it with the cheap one first and then escalate to a larger model if it fails. You’ll often find that scaling up is a lot easier than scaling down in terms of model intelligence.

The takeaway here is that cost and latency belong in or alongside quality in an eval harness.

Once you start recording all three, then you can start measuring and deciding between models.

Your Vibes Are Not an Agent Eval

Fri, 07 Aug 2026 10:00:00 -0500

Do you really know what you’re doing? You swapped a model, you tuned an agentic workflow, you had the agent rewrite a chunk of a system prompt. You added a skill. Now the output feels sharper. Is that feeling a measurement? No, it’s an impression. This is when the vibes start seeping into your agentic engineering world view.

How Impressions Fail

A subjective assessment isn’t necessarily useless, since it’s how you can notice something is wrong in the first place, and it can be a good signal, or an early signal that leads to a corrective action.

A few things here conspire to work against you.

Recency. You remember the last three runs vividly and the forty before them not at all. If the last three happened to be easy tasks, the model got better. If they were gnarly, it got worse.

Confirmation. You just spent an hour rewriting a prompt. You are not a neutral judge of whether that hour helped. Nobody is.

Prompt drift. This one is sneakier. You’re not asking the same thing you asked last month. Your prompts got better because you got better at prompting, and that improvement gets silently credited to the model.

Task drift. The work changed. You were doing greenfield scaffolding in June and you’re doing debugging in August. Those exercise completely different capabilities, and comparing across them tells you nothing.

All of these will sneak up and bite you in the ass. A decent working knowledge of the system is not a measurement.

The actual risk with an agentic workflow isn’t a sharp and dramatic decline in quality. It’s a slow regression over time as you start missing things that slip through the cracks when you’re not paying attention as closely as you should on that day.

I talked about evals that are worth building in previous posts. You should go have a look at some examples there on how to get started.

How do you test a harness? You need to separate the model from the harness. It turns out the harness changes frequently along with the model. Is it even worth testing the harness?

A model swap tripwire is a good place to get started. A tripwire asks whether this specific change made things worse. It’s a binary operation. You run it before the swap, save the results, and run it again after the swap, and compare the results. Same task, same prompt, only the model changed.

If you keep going on vibes, they will keep telling you things. That may or may not matter. At the end of the day, vibes are a decent smoke alarm, but make for a terrible way to measure quality.

Start With Ten Tasks You Actually Do

Thu, 06 Aug 2026 10:00:00 -0500

Public benchmarks of large language models are a fine way to compare models in the abstract, but they’re close to useless for answering questions about things that actually matter. Generally, it’s helpful to know which model is the best in general at a specific benchmark, but it doesn’t answer the question of which model is the best at that specific thing you ask it to do all day long.

You need to build a golden set. Ten tasks where you understand the input and the output.

Scoring the code is not the hard part. Picking the ten tasks is going to be the hardest thing. How do you pick something that has a true pass or fail, but also applies to your specific problem?

The temptation is to sit down and try to come up with representative tasks. Chances are you’re going to waste a ton of time and not produce any better results.

Instead, you should be harvesting your tasks from a variety of sources.

What happens when the benchmark that matters is your last ten pull requests?

Git history. What have you actually been changing? A month of commits will show you the shape of your work faster than introspection will.
Issues and TODOs. These are tasks somebody already wrote down in task-shaped language.
Prior agent sessions. If you have logs, this is the best source, because it’s literally the distribution you’re trying to measure.
The things you retry. Anything you’ve asked an agent twice because the first answer was wrong is a high-value task. It’s already demonstrated it can discriminate.

A good eval task is one where you already know what the failure is and that it’s possible, because you’ve seen it fail.

Scoreable Means Checkable

So now you have your tasks, and it’s a different problem. You need to decide how to score the task and whether or not the agent got it right. This doesn’t necessarily mean you have to build automation from day one. Some good starting points that would qualify are the following:

Tests pass or don’t
Output parses or doesn’t
The right files changed and no others
A required field is present and well-formed
The result matches a known-good output you saved earlier

Keep the amount of comparisons small. Don’t expand and keep evaluating. You can keep your old evaluations, but they shouldn’t impact future decisions forever.

Some amount of change in passing or failing over time is representative of a healthy set.

Here’s how you can get started.

Open your git log. Find things that the agent did well and things the agent could have done better. Write a definition of done.

You now have a small golden set that’s going to be more relevant and useful than any leaderboard online, because it was built from the results of your actual work.

The leaderboard tells you which model wins on average. You are not the average.

`updated_at` Is Not a Conflict-Resolution Strategy

Wed, 05 Aug 2026 10:00:00 -0500

In the last post we talked about the problems with a distributed system, and touched on the fact that timestamps are not as reliable as you think they are.

If you have two updated_at fields and you compare them, how do you decide which side is the correct one?

The updated_at field only tells you that a write happened. It doesn’t tell you the meaning, or if it was intentional. Conflict resolution is fundamentally a question about causality. Did side A intend for this change to happen? A wall clock timestamp can’t tell you the answer to that.

Two independent clocks can drift, and will drift. Yes, it will get corrected by NTP occasionally. But you can’t always rely on their NTP service working. Timestamps are a fine signal that something occurred, and they’re a reasonable way for a human to sort a list and answer roughly when we think a change occurred. But if you use them as a foundation to decide what data to keep, you’re gonna end up destroying and losing data.

Things That Actually Work

The good news is the alternatives are not exotic, and you don’t need all of them.

Content hashes. Hash the meaningful content and compare hashes instead of times. This kills the metadata-edit problem outright: if the hash matches, nothing changed, no matter what the timestamp claims. It’s the highest-value change on this list and usually the easiest, because it’s a pure function of data you already have.

Version counters. A monotonic integer per record, incremented on every meaningful write. Immune to clock skew entirely, because it isn’t a clock. The cost is that somebody has to own the increment, which is straightforward with a single authority and gets harder without one.

Sync checkpoints. Record what was confirmed at the last successful sync, not just when it happened. Then the question becomes “has this changed since the last agreed state,” which is answerable, instead of “is this newer,” which is a guess.

Operation logs. Store what happened rather than only the result. Heavier, but it’s the only option that lets you reconstruct intent after the fact, and it turns “which one wins” into a question you can actually audit.

You can get most of the benefit from the first one. Hash the content, and let the timestamp go back to being a display field.

When Last-Write-Wins Is Fine

I’m not gonna lie, last write wins is often the correct engineering choice, and replacing it with something more complicated can be its own mistake. Sometimes it’s fine. If a write gets lost and the data is recoverable, that’s a trade you can live with.

If it’s a simple tool without a ton of users, adding a lot of complexity is not the way to go.

If the data is just a cache or a projection, then who cares? You can rebuild it from the authoritative source anyway.

What I’d Actually Do

Keep updated_at. It’s useful. Sort by it, display it, log it.

Just stop letting it decide things. Add a content hash and check that first, so a no-op edit stays a no-op. If a field can be written from two sides independently, give it a version counter or an explicit authority rule, and write the rule down somewhere the next person will find it.

The Moment You Add Sync, You Have a Distributed System

Tue, 04 Aug 2026 10:00:00 -0500

How do you keep two sets of data in sync? Like, by definition, you now have a distributed system.

It could be something simple, syncing files or talking with a remote service somewhere. Maybe it’s not a lot of code. Initially, it might not feel like a distributed system, because there’s no cluster or consensus protocol. There’s no leader election system. You have multiple leaders that need to stay in sync.

How do you maintain state between two independent systems, when their only connection is an over-the-network connection that’s allowed to fail?

Sync can be a verb that you apply to the data on one side, but it’s also describing the negotiation that happens between two distributed systems.

Here is the set of questions you have to answer if you are trying to build a distributed system that maintains sync.

What’s new here that isn’t there?
What’s new there that isn’t here?
What changed in both places since we last talked?
What happens if we get halfway through and the connection dies?
If I retry, do I create a duplicate?

All of these sound like problems from a paper you read about replicated state machines. Congratulations, they’re now your problem too.

Just because the request succeeded doesn’t mean that the two systems now agree.

The problems that you’re going to run into are either caused by or solved by a timestamp.

Recovering from an error state is crucial for building a durable system.

Idempotency Is the Cheapest Insurance You Can Buy

It is a guarantee, or pretty much a guarantee, that if your sync can be interrupted, it will be. Idempotency is how you ensure that the same request can be retried safely. If you run the same request twice, you either need to produce the same result or no result.

Every item on both sides of the system needs its own stable identity that each side agrees on. The create needs to always be create-if-absent, basically an upsert.

How do you decide which copy of the data has authority?

When you start needing to do resolution logic, this is where your subtle data loss can occur. Your point-in-time recovery window is likely 30 days or less, and chances are you aren’t going to go back and check the old copy that is about to expire.

Last write wins is the default because it’s easy, and it’s what everybody assumes. It works when there aren’t a whole lot of writes and when one side is clearly the primary. It breaks down when the data can’t be replayed safely.

Do your timestamps actually mean what you think they mean? Can you trust time? It’s complicated to get correct. And if the difference between two timestamps is very small, and the drift is larger than the difference, problems occur.

Things to look into for later: CRDTs, vector clocks, operational transforms.

Chances are these are not the right answer for a personal tool that you’re building on the weekends. It’s good to have discipline and understand the solutions we’ve come up with for resolving synchronization problems. At the end of the day, you’re just gonna want something that works.

You have a distributed system. It has one user and it runs on a laptop, but it has all the failure modes, and it doesn’t care that you didn’t mean to build one.

Your Local File Should Not Have to Argue With Your Database

Mon, 03 Aug 2026 10:00:00 -0500

Sync bugs usually all start the same way. Two copies of something, both of them mostly right, and no written rule about which one wins.

The problems occur when you don’t notice The bug. When the file says one thing and the database says another. It’s not a problem until it is. And then you have to spend time figuring the why and when’s of the drift.

So let’s talk about authority. Not storage, not sync, not “where does the data live.” Authority. Which copy is allowed to be right when two copies disagree.

The Question Nobody Writes Down

Most systems that hold the same data in two places never actually decide this. The decision gets made accidentally, by whichever code path happened to run last, and then it gets re-made differently by the next feature.

The failure is duplication without a stated rule.

Here’s a concrete version. I run a content pipeline for this blog. Posts are Markdown files with YAML frontmatter sitting in a directory. There’s also a Turso database holding metadata about those same posts. Two copies of what looks like the same information.

Ask the naive question, “which one is the source of truth,” and you get a bad answer, because the honest answer is neither, and both, depending on the field.

Split Authority by Field, Not by Store

You might try to pick an authoritative source based on store. Files win, or the database wins. But the useful granularity is usually the field.

In my pipeline it breaks down like this:

Post content and tags: the Markdown file wins. The frontmatter is authoritative. If the database has a different tag list, the database is wrong, and it gets rebuilt from the file.
Scheduling: the database wins. What time a post goes out, what slot it holds, whether it’s been claimed. The file does not get a vote.

Those are different answers for the same post, and that’s fine, because each one is written down and each one has a reason.

The content lives in the file because content is the thing I edit by hand, in an editor, with Git history behind it. I want git log to be the real record of what changed. Putting that in a database would mean my writing history lives somewhere that is harder to access.

The schedule lives in the database because scheduling is a coordination problem. It needs uniqueness constraints, it needs to answer “what’s in the 10am slot on Tuesday,” and it needs to do that without me parsing 241 files. A database is genuinely better at that. It just isn’t better at holding prose.

A Database Can Be Useful Without Being Authoritative

I think there’s a reflex where adding a database feels like promoting the data into it. You put the posts in Postgres and now Postgres is where posts are.

It doesn’t have to work that way. A database can be a query layer over data that lives somewhere else, and that’s a completely respectable job. Indexes, joins, counts, “show me every post tagged local-first published before June.” All of that is worth having, and none of it requires the database to be the authority.

The test I use: if I deleted the database right now, what would I lose forever?

For me, it would be the scheduling state because that’s what I put in the database. The important thing is I wouldn’t lose a single word that I’ve written. Every post would still be in a directory. This choice is deliberate.

It’s easy for the database to become a Cache and not an authoritative source.

What Should Happen When They Disagree

If you have documented your authoritative source, then the disagreements stops becoming a crisis, and it just is a routine. Resolution event

You should be able to rebuild it. There should be nothing to decide. The decision is documented and how you resolve conflicts. Just depends on. Which authoritative source owns Which s segment of your data?

In my case, there’s actually a third authoritative source, and that’s the remote blog system that hands back an ID every time I schedule a new post.

So, this is totally fine if you pick the authority at the field level and Document that decision to prevent trip-ups in the future.

Your files and your database shouldn’t be arguing, all it requires is a bit of planning.

The Decision Log: A Lightweight Artifact for Agentic Coding

Sun, 02 Aug 2026 10:00:00 -0500

Coding agents are remarkably good at reopening decisions you already made.

Imagine a content pipeline where posts live as local Markdown files and a database holds the scheduling metadata. You open a fresh session. Which one does the agent think is authoritative?

It has to guess. And the database looks like the better answer, because databases usually are. So it proposes the obvious cleanup: make the database the source of truth and treat the files as an export format.

It’s wrong, for a reason the code never states. You can rebuild the database from the files. You cannot rebuild the files from the database. Drop the database and you re-index from markdown, losing some operational history on the way. Lose the markdown and the posts are gone.

Some questions are still better answered by the database, scheduling among them. But the authoritative copy is whichever one you could regenerate the other from, and that asymmetry is the entire argument. It appears nowhere in the schema, nowhere in the file layout, and nowhere in the diff.

So you explain it. The agent takes the point immediately, drops the idea, and gets on with the actual work. Good outcome.

Then the session ends, and the next one arrives with the same instincts and the same blank slate. Nothing’s wrong with its reasoning. It’s missing the one piece of context the repository doesn’t contain: that road has already been walked.

That’s what a decision log is for.

Issues and Skills Leave a Gap

Agentic projects tend to accumulate two useful artifacts.

Issues describe work. Add a field, fix the retry logic, move the cache, update the API client. They tell an agent what needs to change and, if the issue is any good, what done looks like.

Skills and project instructions describe process. Use this package manager. Run these tests. Never call this destructive command. They tell an agent how work should happen, repeatedly.

Both matter. Neither is a natural home for, “We considered Redis, rejected it because this service has to run without another dependency, and we’ll reconsider if the process moves to multiple instances.”

Artifact	The question it answers
Issue	What needs to change?
Skill or instruction	How should work be done?
Decision log	Why this path instead of another one?

The code records the result. Git records the diff. A closed issue might contain the discussion, if somebody thinks to go looking for it. The decision log keeps the conclusion and the rejected alternatives somewhere an agent can find them before it starts planning.

The Rejected Option Is the Important Part

I wrote recently about decision debt, the gap created when code gets produced faster than anyone records what it means or why it exists. A decision log is one small way to pay that debt as you go.

The useful part isn’t “We chose SQLite.” The repository already contains a SQLite database. The useful part is why SQLite won, which alternatives lost, and what would have to change before the decision should be reopened.

Without that negative history, an agent sees an absence and treats it as an oversight. No Redis? Add Redis. No abstraction around this HTTP client? Generate one. No Kubernetes deployment? Surely the project just hasn’t matured enough yet.

Sometimes the missing thing is missing on purpose.

Humans do this too, of course. We reopen old arguments when the people who remember them leave, or when the conclusion is buried in a meeting recording. Agents just compress the cycle. Every fresh session is a new developer joining the project with excellent technical instincts and absolutely no institutional memory.

Keep the Entry Small

Architecture Decision Records have been around since Michael Nygard described the pattern in 2011. They preserve the status, context, decision, and consequences of an important architectural choice. Lightweight templates like MADR also capture the options considered, the decision drivers, and why one option won.

That’s the right idea. I just don’t need a formal architecture record for every consequential choice in a personal project.

I’d start with some sort of log file, holding entries like this:

## 2026-07-26: Keep Markdown as the publishing source of truth

**Context:** Posts exist as local files and as database records.
**Decision:** Frontmatter controls publish state. The database owns
scheduling metadata and is not authoritative for publishing.
**Rejected:** Making the database authoritative, or newest-write-wins.
**Why:** The database can be rebuilt from the files. The files cannot be
rebuilt from the database.
**Revisit when:** Editing moves to a multi-user hosted application.
**Links:** the repository module, the publishing documentation.

That’s the whole artifact.

An agent that reads that before it starts planning knows the database-as-source-of-truth idea isn’t a fresh insight. It also knows exactly what would have to change before it becomes one again.

I’ll be honest that I haven’t started doing this yet, but it sounds like a good idea right?

The entry doesn’t need a transcript of the debate. It needs enough context for a future person or agent to understand that the alternative was considered, why it lost, and which changed condition would make it worth discussing again.

What Belongs in the Log

If every choice becomes an entry, the log turns into another file nobody reads. I’d record a decision when at least one of these is true:

Two or more reasonable approaches existed
A future agent is likely to propose the rejected option again
The choice establishes a source of truth, security boundary, schema, dependency, or workflow
Reversing it later would be expensive or dangerous
The reason isn’t obvious from the code

Don’t log naming arguments, routine implementation details, or every library function you picked. “Used a dictionary here” isn’t institutional knowledge. “Kept provider integrations on raw HTTP because the SDK doesn’t support the endpoint we need” might be.

Major decisions can still become full ADRs in docs/decisions/ or whatever. The lightweight log is for the big middle ground between an architectural record and a comment somebody vaguely remembers leaving on a pull request.

Make Agents Read It at the Right Time

Creating the file isn’t enough. The agent needs a retrieval rule. One sentence in the project instructions:

Before proposing changes to architecture, dependencies, data ownership,
security boundaries, or core workflows, search `docs/decisions.md` for
related decisions and revisit conditions.

Then the other half:

After a consequential decision is approved, propose a short decision-log
entry. Do not record a new project policy without human confirmation.

The instruction defines the recurring behavior. The log supplies the project-specific facts. That keeps settled choices out of a giant instruction file while still putting them in the agent’s path when they matter.

When a decision changes, don’t quietly rewrite history. Add a new entry that supersedes the old one and say which revisit condition actually showed up. Version control preserves the edit, but the document should make the change legible without requiring repository archaeology.

Not Another Memory System

A chat transcript contains every false start, tool result, and half-formed idea. It’s too noisy to act as a project constitution. Agent memory can help retrieve past context, but it might be private to one tool, unavailable to a collaborator, or hard to review in a pull request.

A decision log needs to be deliberately boring. Plain text. Searchable. Reviewable. Stored next to the code, or accessible by it. Any human or agent can read the same entry and argue with it in the open.

None of this guarantees an agent will make the right call. It removes one wasteful failure mode: spending another session rediscovering a settled tradeoff and confidently proposing the option the project already rejected.

Issues tell the agent where to go. Skills tell it how to move. The decision log marks the roads we already closed, why we closed them, and when they might be worth opening again.

That feels like context worth keeping.

Sources

Documenting Architecture Decisions — Michael Nygard’s original 2011 ADR proposal.
MADR — a lightweight decision-record format covering options, rationale, consequences, and revisit conditions.
The GDS Way: Documenting architecture decisions — keeping decision rationale in the repository while using issues to track implementation.
How Claude Code works — fresh session context, compaction, project instructions, and persistent memory.

Run Your Whole Agent Stack on a $5 Box

Sat, 01 Aug 2026 10:00:00 -0500

I SSH’d into my home server this afternoon and ran docker stats on the memory layer that every one of my coding agent sessions talks to. Here’s what came back:

mem0-qdrant    28.09MiB / 60.75GiB    2.13%
mem0-neo4j     612.7MiB / 60.75GiB    0.77%

640 megabytes. Vector store and graph store, both up for three weeks straight, serving every remember and recall call my agents make. The entire persistent memory for my AI tooling uses less RAM than one Chrome tab with Figma open.

So let’s talk about why you’re paying a monthly subscription for this.

Local-First, Not Local-Only

I want to be precise here, because “self-hosted AI” has become a phrase people use to mean nine different things.

My setup is local-first, not local-only. The state lives on my hardware. The memories, the embeddings, the graph relationships, everything my agents have learned about my projects, all of it sits on a box I own, in a Docker volume I can tar and carry away. Nobody can deprecate it, price-hike it, or sunset it.

The inference does not. My embedder points at Mistral’s managed API. I’ll get to why, and how to swap it, but I’m not going to pretend otherwise in a post about self-hosting.

That embedder is the only thing that leaves my network, and only when something actually gets embedded, so when writing a memory and searching for one. Listing, deleting, and every graph operation are local with zero API calls.

State is what you can’t get back. Compute is a commodity you rent by the token. Losing access to an API means switching providers. Losing two years of accumulated project context means starting over.

The Four Pieces

Qdrant is semantic search. When I ask what it remembers about my package manager preferences, Qdrant turns that into a similarity query and hands back the relevant memories. It’s Rust, it’s fast, and at 28MB resident it’s essentially free to run. One gotcha: vector dimensions are fixed when the collection is created. Swap embedding models and you need a new collection, not a migration. I learned that the annoying way.

Neo4j is the graph store. Vectors are great at “find me things that sound like this” and bad at “what depends on what.” The graph holds explicit subject-predicate-object facts, so project X built_with Python 3.13 is a traversable edge instead of a fuzzy match. It’s the heavy one at 613MB, but it’s a JVM, so that’s mostly heap floor rather than working set. If you’re squeezing onto the smallest possible VPS, interrogate this one first.

mem0 is the orchestration on top: what gets extracted from a conversation, what gets deduped against existing memories, what gets written where. That’s the difference between a database and a memory system.

The MCP server is what makes any of it useful. A small Go binary that speaks Model Context Protocol over stdio to Claude Code, exposing eight tools: remember, recall, list_memories, forget, memory_stats, add_relation, recall_related, forget_relation.

The topology is deliberately boring:

Mac                                    Home server
┌──────────────┐   ┌──────────┐        ┌─────────────────┐
│ Claude Code  │◄─►│ mem0-mcp │  LAN   │ Qdrant + Neo4j  │
│              │   │ (Go)     │───────►│                 │
└──────────────┘   └──────────┘        └─────────────────┘
       stdio                    HTTP + bolt

Client binary on my laptop, containers on a box. No cloud in the middle, no account, no dashboard, no seat license.

The Ansible Role Is the Whole Argument

Anyone can docker compose up a stack once. That’s a weekend, not infrastructure. What makes this real is that it’s a role in a repo, and rebuilding it on a fresh box is one command:

ansible-playbook -i common_hosts home.yml --tags mem0

That role does the unglamorous work:

Installs a read-only deploy key scoped to exactly one repo, with an SSH Host alias so it can’t collide with my personal GitHub key
Clones and updates the source at a pinned branch
Templates a .env with secrets pulled from Ansible Vault, no_log: true so nothing leaks into terminal output on a --diff run
Brings up the compose stack with remove_orphans: true, so when I dropped a service upstream, the stale container went with it instead of lingering forever

Be careful with your secrets and how you are creating your .env files!

The Honest Part About the API Key

I self-host the state and rent the inference. Two reasons.

The first reason is speed. I ran embeddings locally before this, on CPU, and it was miserable: roughly 87 seconds to embed 32 memories, against about 2 seconds through a hosted API. That is a 45x difference on an operation sitting directly in the path of every remember and recall. A good model on a CPU is still a slow model, and this was never a quality problem.

The second is that embeddings have gotten cheap enough that not worth the time to setup your own embedding service. Mistral charges $0.10 per million tokens for mistral-embed. Google’s gemini-embedding-001 is $0.15 per million, halved on their batch API. Both are good models. Both bill you.

Cloudflare is the worth knowing about if you’d rather not pay at all. Workers AI includes 10,000 neurons per day free, on the free plan as well as the paid one. Neurons are their normalized compute unit, and bge-m3 costs 1,075 of them per million input tokens — so that daily allowance is roughly nine million tokens a day, at no cost. Past it you’re at $0.012 per million, which is an order of magnitude under the paid competition. For a personal memory layer, nine million tokens a day is not a trial. It’s just free.

One detail if you’re swapping: bge-m3 emits 1024-dimension vectors, the same as mistral-embed. Go back to that Qdrant gotcha — matching dimensions means your existing collection still works. Mismatched ones mean starting over.

And the escape hatch is already built. The env vars in my role are TEI_BASE_URL, TEI_MODEL, TEI_DIMENSIONS. Generic OpenAI-compatible embedder knobs, named after Text Embeddings Inference for historical reasons and pointed at Mistral today. Aim them at a self-hosted TEI container, at Ollama, at anything speaking that shape, and the rest of the stack doesn’t notice.

That’s what local-first buys you. Not purity. Optionality.

So, the $5 Box

My server has 60GB of RAM, which is absurd overkill and exists because it does a dozen other things. The stack itself measured 640MB with three weeks of uptime, essentially zero CPU at idle.

That fits comfortably on a small cloud VPS in the few-dollars-a-month range. Check current pricing yourself rather than trusting a number in a blog post, but the shape is: a 2 vCPU / 4GB instance from Hetzner or similar costs less per month than one seat of most AI memory SaaS products, and you get to run everything else on it too.

Your real constraint is RAM, specifically Neo4j’s JVM floor. On a 1GB instance you’d be fighting it. At 2GB you’re fine. At 4GB you’ll forget it’s running.

Why I Care

The indie web ethos is about noticing that renting your identity from a platform means the platform decides what happens to it.

We’re about to make the same mistake with agent memory, except worse, because the thing being accumulated this time is a working model of how you think and what you’re building. Every “our AI remembers you across sessions” product is a proposal that you deposit that into someone else’s database and hope the pricing page stays reasonable.

Qdrant is Apache 2.0. Neo4j Community is GPL. Docker Compose is a YAML file. Ansible is idempotent YAML. Nothing in this stack is exotic. The barrier to owning your agent memory is an afternoon and 640 megabytes.

Not everyone needs this, and I’m not going to pretend a solo dev with three side projects is being exploited by a $20 subscription. But if you’re accumulating context you’d be genuinely sad to lose, the math changes. Own the state, rent the compute, and keep the role in version control so the whole thing is reproducible on a box you haven’t bought yet.

Moving the embedder onto Cloudflare’s free tier is next on my list, what’s on yours?