security

Two SQLite Libraries, One Deleted Password
Every format in this series so far has been a way of writing a document down. SQLite is the one that asked a different question: what if the application file format were a database?

It is the best-designed thing I have written about in this whole series. It has a real specification, a versioned header, a page allocator, transactions, and a compatibility promise going back to 2004. The Library of Congress recommends it for long-term preservation. Its own documentation makes the case that a .sqlite file is a better application format than a custom binary blob or an XML tree, and the case is correct.

And it will still hand a deleted password to anyone who runs strings on the file, or not, depending on which copy of the library you happened to link.

The Header Is Legible

The first hundred bytes of any SQLite database are a fixed-layout, big-endian header. You can read it with xxd:
```
$ xxd -g 1 -l 32 notes.db
00000000: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00  SQLite format 3.
00000010: 10 00 01 01 00 40 20 20 00 00 00 06 00 00 00 02  .....@  ........
```
Sixteen bytes of magic, and it is the literal string SQLite format 3 with a nul terminator. Then decoding what follows:
```
page size        4096
write version    1  (1 = rollback journal, 2 = WAL)
read version     1
reserved/page    0
change counter   6
size in pages    2
freelist head    0
freelist count   0
text encoding    1  (1 = UTF-8)
sqlite version   3047001
```
The page size is a two-byte field, which creates a small problem: the format allows 65,536-byte pages, and 65,536 does not fit in two bytes. The spec’s answer is a special case, and it is worth reading in the original:

to specify a 65536-byte page size, the value at offset 16 is 0x00 0x01. This value can be interpreted as a big-endian 1 and thought of as a magic number to represent the 65536 page size.

That is the whole hack, documented in the spec, with the word “magic number” used by the authors about their own file. Compare that to every other format in this series, where the ambiguities were things nobody wrote down.

Every page after the header starts with a one-byte type flag:
```
page 1: flag 0x0d  leaf table
page 2: flag 0x0d  leaf table
```
0x0d is a table leaf, 0x05 a table interior node, 0x0a an index leaf, 0x02 an index interior node. Four values, and from those four you can walk a B-tree without a library. Each page is slotted: a pointer array grows down from the header, cell content grows up from the bottom, and free space is whatever is left in the middle.

This is a database engine’s internal layout, written down, versioned, and promised to stay readable. There is a reason people keep reaching for it as a document format. The project’s own argument for it is that you get transactions, partial reads, and incremental writes for free, and that for blobs under about 100KB it reads and writes faster than the filesystem does.

Two Libraries, Two Answers

Here is a three-row table. I delete the row with the secret in it, and then look at the file.
```
import sqlite3
c = sqlite3.connect("notes.db")
c.executescript("""
CREATE TABLE notes(id INTEGER PRIMARY KEY, body TEXT);
INSERT INTO notes(body) VALUES ('the wifi password is hunter2-CANARY');
INSERT INTO notes(body) VALUES ('groceries: milk, eggs');
INSERT INTO notes(body) VALUES ('call the dentist back');
""")
c.commit()
c.execute("DELETE FROM notes WHERE body LIKE '%CANARY%'")
c.commit()
```
```
$ strings notes.db | grep -i canary
(Sthe wifi password is hunter2-CANARY
```
The row is gone from every query. SELECT count(*) returns 2. The bytes are still on the page, because deleting a row marks its cell as a freeblock and links it into a chain of free space. Nothing overwrites it. The next insert that happens to fit will land there and clobber it, and until then it sits in the file.

That is the behavior everyone knows about. If you ran the same three statements through the sqlite3 command-line tool on the same machine, on the same directory, and the string will be gone.
```
$ sqlite3 --version
3.51.0 ...

$ sqlite3 notes.db 'PRAGMA secure_delete;'
2
```
```
$ python3 -c "import sqlite3; print(sqlite3.sqlite_version); \
    print(sqlite3.connect(':memory:').execute('PRAGMA secure_delete').fetchone()[0])"
3.47.1
0
```
secure_delete off means freed space keeps its contents. secure_delete set to 2 is FAST mode, which zeroes freed space inside a page without chasing it into the freelist. macOS ships one build with it on. CPython ships another with it off. Neither reports the difference in PRAGMA compile_options. Nothing in the file records which one wrote it.

So “does my database keep deleted data” has no answer at the format level. It has an answer per binary, and both binaries are called SQLite, and both are on this laptop.

VACUUM does resolve it, because it rebuilds the entire file from live rows:
```
$ python3 -c "import sqlite3; sqlite3.connect('notes.db').execute('VACUUM')"
$ strings notes.db | grep -ci canary
0
```
Which is the same shape as yesterday’s PDF post. The safe operation and the obvious operation are different operations. DELETE is the obvious one.

The Single-File Database Is Three Files

The pitch for SQLite as an application format leans hard on it being one file you can copy, email, and back up. In the default rollback-journal mode that is true between transactions. Turn on WAL, which is what you want for anything with concurrent readers, and while a connection is open:
```
-rw-r--r--  4096 w2.db
-rw-r--r-- 32768 w2.db-shm
-rw-r--r-- 12392 w2.db-wal
```
The write-ahead log and the shared-memory index are each larger than the database. The -wal file starts with its own magic number:
```
37 7f 06 82 00 2d e2 18 00 00 10 00
```
0x377F0682 says little-endian checksums, and the last four bytes are the page size again, 4096, matching the main file.

Copy just w2.db while that connection is open and you get the database as of the last checkpoint, silently missing every committed transaction still sitting in the WAL. The main file’s header even tells you this is possible: the read and write version bytes both flip from 1 to 2 when WAL is enabled, which is the format’s way of saying “a reader that does not understand WAL must not open me.”

Both files disappear on a clean close, which is why this bites people at exactly the wrong moment. Your backup script tested fine.

What To Do About It
- Set PRAGMA secure_delete=ON explicitly if the database holds anything sensitive, in your application code, not in a config file you hope got read. Do not assume the default. Check it at startup and log what you got.
- VACUUM after deleting anything that mattered. It is the only operation that actually reclaims and rewrites. On a large database it is expensive and takes a full-file lock, so schedule it.
- Back up with .backup or VACUUM INTO, never cp. Both take a consistent snapshot including the WAL. cp on a live WAL database gives you a file that opens fine and is missing data.
- Check PRAGMA integrity_check in your test suite, not just in production incident response.
- Read the header yourself when debugging. Page size, page count, encoding, and journal mode are all in the first 100 bytes, and a hex dump will tell you in two seconds whether the file is truncated, whether it is even SQLite, and what wrote it.
- Use it as an application file format. Seriously. The argument is right. Just know that “it is a database” means it has a database’s habits, including keeping things you told it to delete.
Two weeks into this series the pattern is uncomfortably consistent. The problem is almost never the specification. CSV had none, JSON had a small one, YAML had a good one that arrived late, XML had one for everything, and SQLite has one that is legible, complete, and honest about its own hacks. It still ships a data-remanence behavior that flips based on a compile-time flag neither binary bothers to report.

The format is not what you get. The build is what you get.

Sources
- SQLite Database File Format — the 100-byte header, the page types, the freeblock chain
- SQLite as an Application File Format — the project’s own argument, including the claim that it beats the filesystem for blobs under 100KB
- Write-Ahead Logging and the WAL file format — the -wal and -shm companions, and why cp is not a backup
- PRAGMA secure_delete — including the FAST mode that macOS ships and CPython does not
- Library of Congress: SQLite as a recommended preservation format — the format description
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 27, 2026 Programming Sqlite Databases security File-formats

Enjoyed this?
Your PDF Remembers What You Deleted
A PDF does not have a current version. It has a stack of them, and the one your reader shows you is whichever one is on top.

That is not a bug, but a clause in the specification, it is the reason PDFs can be signed and annotated without invalidating anything that came before, and it is also why a court filing with black rectangles over the sensitive parts keeps handing those parts to anyone who runs pdftotext.

I built a 619-byte PDF by hand to show the mechanism, because the mechanism is small enough to fit in a blog post.

Read It From the Bottom

Here is the entire tail of that file:
```
xref
0 6
0000000000 65535 f 
0000000015 00000 n 
0000000064 00000 n 
0000000121 00000 n 
0000000247 00000 n 
0000000366 00000 n 
trailer
<< /Size 6 /Root 1 0 R >>
startxref
436
%%EOF
```
A parser opens this file by seeking to the end, finding %%EOF, reading the number above it, and jumping to byte 436. That is the cross-reference table. Every line in it is a ten-digit zero-padded byte offset for one object. Object 4 lives at byte 247. Object 5 lives at byte 366. The trailer then says the document catalog is object 1, and the parser follows the graph from there.

Nothing is read sequentially. There is no “parse the header, then stream forward.” The header is four bytes of version number and a comment containing four deliberately non-ASCII bytes, present only so that any transport handling the file is forced to treat it as binary.

If you read the ZIP post earlier this month, this should feel familiar. Both formats put their index at the end so that appending is cheap. Both formats therefore have the same structural property: the index is authoritative, and the bytes it doesn’t point at are still sitting right there in the file.

The Black Box Is a Rect

Start with the failure, the one that has been in the news.

My page has a single content stream with a text-drawing operator in it:
```
BT /F1 12 Tf 20 60 Td (CONFIDENTIAL: Q3 layoffs begin March 14) Tj ET
```
To redact it, I append a filled black rectangle covering the same coordinates:
```
0 0 0 rg 15 52 270 24 re f
```
On screen that is a black bar. The text is underneath it and completely invisible. Then:
```
$ pdftotext overlay.pdf -
CONFIDENTIAL: Q3 layoffs begin March 14
```
The rectangle is a drawing instruction. Tj is a different drawing instruction. Text extraction never rasterizes anything, so it never learns that one shape happens to sit on top of another. It walks the content stream, collects the text operators, and hands them over.

This is what Paul Manafort’s lawyers filed in January 2019. The blacked-out passages came straight back out with copy and paste, and what came out was that Manafort had shared 2016 campaign polling data with Konstantin Kilimnik. A corrected version went up quickly.

It is also the 93-page TSA screening manual posted to a federal contracting site in December 2009, with black boxes over the tolerances used by airport metal and explosive detectors and over the special handling rules for CIA officers, diplomats, and law enforcement. Same mistake, ten years earlier, and the second time nobody could claim the technique was novel.

Doing It Properly and Still Losing

Now the interesting case. This time I actually replace the content stream. New object 4, no text operator, only the rectangle. The old object stays where it is, because PDF’s editing model appends:
```
$ pdftotext redacted.pdf -
$
```
Empty. The text is gone from the document as the parser understands it. This is an edit, not an overlay, and every viewer will agree the page contains a black bar and nothing else.
```
$ strings redacted.pdf | grep -i confidential
BT /F1 12 Tf 20 60 Td (CONFIDENTIAL: Q3 layoffs begin March 14) Tj ET
```
The file is 812 bytes. The first 616 of them are the original document, untouched, including its own xref, its own trailer, and its own %%EOF. The new revision was appended after that:
```
$ grep -abo '%%EOF' redacted.pdf
613:%%EOF
806:%%EOF
```
Two end-of-file markers in one file. The second trailer carries a /Prev key pointing back at the first cross-reference table:
```
trailer
<< /Size 6 /Root 1 0 R /Prev 436 >>
startxref
695
%%EOF
```
So recovering the unredacted document is not forensics. It is head:
```
$ head -c 616 redacted.pdf > recovered.pdf
$ pdftotext recovered.pdf -
CONFIDENTIAL: Q3 layoffs begin March 14
```
That is a valid, complete, openable PDF. Not a fragment, not a carved string. The original revision was always a self-contained file; the redaction just parked another file behind it.

You can see why the format works this way. Digital signatures need the signed byte range to stay byte-identical, so an annotation or a form fill has to append rather than rewrite. Incremental save is also why a 400-page PDF takes a moment to annotate instead of a minute. The design is coherent. It just means that “save” and “erase” are unrelated operations, and most software offers you the first one while you are thinking about the second.

Hard to Get Right

In 2021 Supriya Adhatarao and Cédric Lauradoux collected 39,664 PDF files published by 75 security agencies across 47 countries and looked at what was still in them. Seven of the 75 agencies had made any attempt at sanitization at all. Of the files those seven had sanitized, 65% still contained sensitive information.

These are security agencies. Sanitizing documents is a thing they have written policies about. The success rate among the small minority that tried was roughly one in three.

The reason is not incompetence. It is that the safe operation and the obvious operation are different operations, and the file gives you no feedback about which one you performed. Both produce a document with a black bar on it.

What To Do About It
- Never redact by drawing. If the feature lives in the same menu as shapes, highlights, and stamps, it is a shape. A real redaction tool deletes the content stream operators; an annotation tool adds one on top.
- Flatten and rewrite the whole file afterward. qpdf --linearize in.pdf out.pdf rebuilds the document as a single revision and drops what nothing points at. If the output still has two %%EOF markers, the collapse did not happen.
- Check your work with strings and pdftotext. Both take ten seconds. Between them they would have caught every failure in this post.
- Count the %%EOF markers on any PDF you receive, not just the ones you send. Use grep -ac '%%EOF' file.pdf; without -a, grep decides the file is binary and reports nothing at all.
- Export to images and re-OCR when the stakes are high enough. It destroys the text layer along with everything else, which is the point.
- Strip metadata separately. Author names, the software that produced the file, and the local file path of the original are in the /Info dictionary and the XMP packet, and none of that is touched by redacting page content.
The formats in the first half of this series failed by being unclear about what their bytes meant. PDF is not unclear about anything. It says precisely where every object is and precisely which ones are current, and it says it in a structure that keeps the non-current ones exactly where they were. The file is honest. It is the word “redacted” that is doing the lying.

Sources
- ISO 32000-1:2008 — the PDF 1.7 specification; clause 7.5.4 covers the cross-reference table, 7.5.6 covers incremental updates
- ISO 32000-2:2020 — PDF 2.0
- Adhatarao & Lauradoux, “Exploitation and Sanitization of Hidden Data in PDF Files” — 39,664 files from 75 security agencies; 7 attempted sanitization, 65% of those still leaked
- Library of Congress format description for the PDF family — history and revision structure
- BuzzFeed News on the Manafort filing — January 2019
- CBS News on the unredacted TSA manual — December 2009
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 26, 2026 Programming security File-formats Pdf Forensics

Enjoyed this?
Every XML Feature Is One You Turn Off
Last post ended on a question. XML is the format in this series that specified everything, so what did all that specifying buy?

An answer for every question, and an attack surface made entirely of answers.

The other formats in this series failed by leaving things out. CSV never had a standard. JSON declined to say what a number means. YAML wrote the rules down and then most implementations kept using the old ones. XML failed the other way. It defined a schema language, a query language, a transformation language, a namespace system, and an entity system, and the entity system will read files off your disk.

Two Kinds of Correct

Start with something XML got right, because it is the only format here that made this distinction at all.

The specification defines two separate bars:

A data object is an XML document if it is well-formed, as defined in this specification. In addition, the XML document is valid if it meets certain further constraints.

Well-formed is syntax. Tags match, there is one root, attributes are quoted. Valid is semantics: the document declares a schema and conforms to it.

CSV has neither concept. JSON has only the first one. XML separated them in 1998, which was ahead of its time, and then almost nobody used the second one. Most XML in production is well-formed and unvalidated, which means it has exactly the same guarantees as JSON with more punctuation.

The Entity System

XML documents are built from entities. Five are predefined, and you know them:
```
&amp;  &lt;  &gt;  &quot;  &apos;
```
You can also declare your own in the document type declaration, which is how you get constants in a config file. And an entity can be declared to pull its content from somewhere else:
```
<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "file:///tmp/xmltest/secret.txt">
]>
<foo>&xxe;</foo>
```
That is a legal XML document. SYSTEM means “go get it.” Here is what a parser does with it:
```
ElementTree  ParseError: undefined entity &xxe;
lxml         XMLSyntaxError: Entity 'xxe' not defined
defusedxml   EntitiesForbidden(name='xxe', system_id='file:///tmp/xmltest/secret.txt')
lxml, resolve_entities=True    'SECRET-CANARY-12345\n'
```
The first three refuse. The fourth read a file off the disk and put its contents in the document tree.

The difference between the second line and the fourth is one keyword argument. Not a patch, not an old version, not a misconfiguration. A parser option, on a current library, named after something that sounds like ordinary XML processing. Of course you want entities resolved. Entities are how the format works.

This is XXE, and it is not a bug in any implementation. Every one of those parsers is behaving as specified. The specification says an external entity is fetched, so the ones that fetch it are correct and the ones that refuse are deliberately non-conforming for your safety.

The Bomb That Got Fixed

The same entity system nests, which produces the XML version of the billion laughs attack:
```
<!DOCTYPE lolz [
  <!ENTITY lol "lol">
  <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
  <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
]>
```
Each level multiplies by ten. A 291-byte file at three levels expands to 3,000 characters. Add two more levels and the arithmetic says 300,000, and the file is still under 500 bytes.

Except it doesn’t, and this is the part worth reporting honestly. At five levels, current libxml2 stops:
```
XMLSyntaxError: Maximum entity amplification factor exceeded,
see xmlCtxtSetMaxAmplification
```
It refuses even with resolve_entities=True. Somebody went and put a ratio limit in the parser, and it works. Compare that to YAML, where safe_load still expands aliases without complaint, and the equivalent 202-byte payload produced 74,732 nodes with no objection at all.

So the XML ecosystem fixed the denial-of-service and left the file-read one flag away. That ordering tells you something about which failure people actually hit.

What the Rest of It Bought

XML shipped an enormous amount of specification, and the pieces are individually good:
- XSD defines 19 primitive datatypes with real inheritance, so <price>12.50</price> can be a decimal rather than a hopeful string. This is precisely what CSV lacks and what JSON refuses to commit to.
- XPath addresses any node in a document without writing a traversal.
- XSLT transforms one document into another declaratively.
- Namespaces let two vocabularies coexist in one file without colliding, which is the problem every format in this series either ignores or solves with a naming convention.
None of that is bad engineering. Read the list again and notice that it is a description of the problems the other five posts were about. XML solved them. In 1998.

The cost was that using XML correctly means knowing which parts to switch off, and the defaults were set in an era that assumed documents came from people you knew. Every hardening guide for XML is a list of features to disable: no DTDs, no external entities, no network access, no schema fetching.

There is a version of this where the lesson is “XML was too complicated.” I don’t think that’s it. The formats that replaced it are simpler and they have the same problems, plus the ones XML had already solved. Nobody misses XSLT, and everybody has now written their own worse version of it.

What To Do About It
- Disable DTD processing entirely unless you know you need it. In Python that is defusedxml; in Java it is disallow-doctype-decl. This closes XXE and entity expansion in one move.
- Never enable resolve_entities on input you did not write. It is the single flag that turns a parser into a file reader.
- Validate against a schema at the trust boundary, not just parse. Well-formed is not a security property. It is barely a correctness property.
- Stream large documents. SAX and StAX read in constant memory; DOM builds the whole tree first, which is its own denial of service if the document is attacker-sized.
- Know your parser’s defaults and pin them explicitly. They have changed over time, usually toward safety, and code that relies on a safe default is one dependency upgrade from a different one.
XML answered every question these posts have raised. It has types, a validation model, a query language, and a namespace system, and it had all of them a decade before the formats that replaced it. What it could not do was make the safe path the default one, and that turned out to matter more than any of the rest of it.

That is the actual pattern across this whole series. Not that formats are underspecified or overspecified, but that the defaults are the specification most people ever use.

Sources
- XML 1.0 (Fifth Edition) — 26 November 2008; the well-formed and valid definitions, and the five predefined entities
- XML 1.0 (First Edition) — 10 February 1998, the original Recommendation
- XML 1.1 — 4 February 2004; the revision almost nobody adopted
- XSD 1.1 Part 2: Datatypes — the 19 primitive datatypes
- OWASP XXE Prevention Cheat Sheet — per-parser hardening settings
- defusedxml — the Python library that turns the dangerous parts off for you
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 22, 2026 Programming security File-formats Parsing Xml

Enjoyed this?
Norway Is Not a Boolean
JSON’s problem is that its specification is too small. It tells you 9007199254740993 is a well-formed number and then declines to say which number.

YAML went the other way. The 1.2.2 specification is a book. It has a formal grammar, a chapter on recommended schemas, and an answer for nearly everything. And it will still read your config file and decide, without asking, that Norway is false.

The Norway Problem

Here is a config file. Every value in it is a string that a human would read as a string.
```
country: NO
duration: 1:20
mode: 0755
```
Three parsers, on those exact bytes:
```
PyYAML 6.0.3   {'country': False, 'duration': 80,     'mode': 493}
ruamel.yaml    {'country': 'NO',  'duration': '1:20', 'mode': 755}
js-yaml        {"country":"NO",   "duration":"1:20",  "mode":755}
```
NO is the ISO 3166 code for Norway. PyYAML returns the boolean false, because YAML 1.1 recognized twenty-two spellings of true and false, and NO is one of them. The spec lists them as a single regular expression:
```
y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF
```
Count them. Twenty-two. Six of those are country codes, single letters, or ordinary English words that appear in real data. Nothing in the file said “this is a boolean.” The parser inferred it from the shape of the text, and the shape of the text was two letters.

1:20 became 80 because YAML 1.1 supported sexagesimal integers, so a duration is read as base 60. One times sixty, plus twenty.

0755 became 493 because a leading zero meant octal. That is a file mode that no longer means what it says.

It Was Fixed in 2009

This is the part that makes YAML different from the other formats in this series.

CSV never had a standard. Markdown had too many. YAML had exactly one problem, everybody agreed it was a problem, and the working group fixed it. YAML 1.2 arrived in 2009 and threw all of it out. Base 60 is gone. Implicit octal is gone. The Core schema recognizes true and false and their case variants, and nothing else.

Seventeen years later, the two YAML 1.2 parsers above return strings, and PyYAML returns False.

PyYAML implements YAML 1.1. It is the default YAML library for Python, it is what pip install pyyaml gives you, and the specification it implements was superseded when the iPhone 3GS was current. The fix exists. It shipped. Most of the ecosystem simply stayed where it was, because changing the type of NO in a minor release breaks every config file that relied on it.

A format can be fixed and still be broken, if the fix arrives after the implementations do.

Everything Else That Isn’t a String

The country-code case is famous. It is not the only one, and the rest are quieter:
```
version: 1.10   ->  1.1        (float, and .10 became .1)
build: 010      ->  8          (octal)
port: 8080      ->  8080       (int, fine, until you concatenate it)
answers: [y, n] ->  ['y', 'n'] (strings)
answers: [yes, no] -> [True, False]
```
The first one is the one that should bother you. A semantic version of 1.10 parses as the float 1.1, which is a different version, and it does it silently in a file whose entire job is to record which version you meant.

And note the last two lines. y and n stay strings in PyYAML while yes and no become booleans, because PyYAML’s resolver implements a narrower set than the 1.1 spec’s regexp advertises. So the answer to “does this parser coerce single letters” is neither yes nor no. It is “some of them, and you have to test.”

Two Ways to Weaponize the Convenience

YAML has anchors. You define a node once with &name and reference it with *name. It is a useful feature for config files with repeated blocks, and it composes.

That is the problem. It composes exponentially.
```
a: &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
```
That file is 202 bytes. Expanding it produces 74,732 nodes, of which 59,049 are copies of the string lol. Add one more line and multiply by nine. This is the billion laughs attack, and the important detail is that safe_load does not stop it. Aliases are not a dangerous tag, they are a core language feature working as designed.

The second way is tags. YAML can annotate a node with a type, and PyYAML historically honored tags that construct arbitrary Python objects:
```
!!python/object/apply:os.system
args: ['id']
```
yaml.load() on untrusted input would run that. It became CVE-2017-18342, CVSS 9.8, published June 2018, with a description that is unusually blunt for the genre: “In PyYAML before 5.1, the yaml.load() API could execute arbitrary code if used with untrusted data.”

The fix took two releases and three years. PyYAML 5.1 deprecated the unsafe default in March 2019. PyYAML 6.0 finally made the Loader argument mandatory in October 2021, so the dangerous call stopped being the short one:
```
>>> yaml.load('a: 1')
TypeError: load() missing 1 required positional argument: 'Loader'
```
The vulnerability was published in 2018. Making the unsafe call harder to type than the safe one landed in 2021.

What To Do About It
- Quote anything that isn’t obviously a number. Country codes, versions, file modes, git SHAs, anything a human would call an identifier. Quoting is never wrong.
- Know which YAML version your parser speaks. If it is Python, assume 1.1 and the Norway problem unless you chose otherwise. ruamel.yaml gives you 1.2.
- Never call yaml.load on input you did not write. safe_load, always. On PyYAML 6 the language makes you say which you meant, which is the correct design.
- Bound the input. safe_load is not a defense against alias expansion. If you parse YAML you did not author, cap the document size before it reaches the parser.
- Use a schema. The value of a schema here is not validation, it is that it declares the type instead of letting the parser guess it from the characters.
YAML’s failure is the opposite of JSON’s, and it produces the same result. JSON declined to say what values mean, so implementations disagreed. YAML said what values mean in enormous detail, got it wrong in 2005, corrected it in 2009, and the correction never fully landed.

Next in this series is XML, which is the one format here that did specify everything. It has a schema language, a query language, a transformation language, and a namespace system. It is worth asking what all of that bought.

Sources
- YAML 1.2.2 Specification — October 2021; the schemas chapter and the rule that tabs “must not be used in indentation, since different systems treat tabs differently”
- YAML 1.1 Boolean type — the twenty-two-form regexp, working draft dated 2005
- CVE-2017-18342 — the yaml.load() RCE, CVSS 9.8
- PyYAML CHANGES — 5.1 (2019) deprecated the unsafe default, 6.0 (2021) made Loader required
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 21, 2026 Programming security File-formats Parsing Yaml

Enjoyed this?

ZIP Files Are Read Backwards

Every format in this series so far reads front to back. PNG starts with a signature and you walk chunks in order. A text file is bytes from the beginning. Markdown parsers scan line by line, top to bottom.

ZIP reads backwards. The index is at the end of the file, and a reader is expected to seek to the end first and work its way back.

That one decision explains almost everything strange about ZIP, including a few things that look like bugs and one thing that is definitely a bug.

The Index Lives at the End

Here’s a real ZIP containing two small text files. 241 bytes total. Scanning it for the four-byte record signatures gives the whole layout:

offset  bytes         ascii   record
     0  50 4b 03 04   P K . . Local File Header      <- hello.txt
    53  50 4b 03 04   P K . . Local File Header      <- second.txt
   108  50 4b 01 02   P K . . Central Directory Header
   163  50 4b 01 02   P K . . Central Directory Header
   219  50 4b 05 06   P K . . End of Central Directory

Every one of those starts with the same two bytes. 0x50 is decimal 80, which is P in ASCII. 0x4b is decimal 75, which is K.

PK. Phil Katz, who wrote PKZIP in 1989, put his initials in the first two bytes of every structure in the format, and they are still there in every .docx, .jar, and .epub on your machine.

The two bytes after PK are the record type: 03 04 for a local file header, 01 02 for a central directory entry, 05 06 for the end-of-central-directory record. Those aren’t printable characters, which is deliberate. A four-byte constant made of two readable letters and two control bytes is unlikely to appear by accident in text, and easy to spot by eye in a hex dump.

The last 22 bytes are the End of Central Directory record, and it’s the entry point:

50 4b 05 06 00 00 00 00 02 00 02 00 6f 00 00 00 6c 00 00 00 00 00

signature            0x06054b50
total CD records     2
central dir size     111 bytes
central dir offset   108
comment length       0

A reader opens the file, jumps to the end, finds that record, reads “the index is at offset 108,” seeks there, and reads the catalog. Listing the contents of a 4 GB archive touches a few hundred bytes.

Note that each file appears twice: once as a Local File Header immediately before its compressed data, and once as an entry in the Central Directory at the end. Hold that thought.

Why Would You Do This?

Because in 1989 you were writing to a floppy disk, and often to a floppy disk that wasn’t big enough.

If the index goes at the front, you have to know everything about every file before you write the first byte: how many files, how big each one compresses to, where each one lands. That means compressing everything to a temporary location, then writing the header, then copying it all back. On a machine with 640K of RAM and two floppy drives, that’s brutal.

Put the index at the end and you can stream. Compress a file, write it, remember where it went. Compress the next one. When you run out of files, write down everything you remembered. One pass, no temporary copy, and you never needed to know the total size in advance.

TAR solved the same problem by having no index at all, which is why tar has to read an entire archive to find one file, and why you cannot randomly access a .tar.gz. ZIP got both streaming writes and random-access reads. That’s the trade that made it win.

The Backwards Scan Is Fuzzier Than It Sounds

The EOCD record is 22 bytes, so you’d think a reader could just read the last 22 bytes and be done.

It can’t, because the record ends with a variable-length archive comment of up to 65,535 bytes. The signature isn’t at a fixed offset from the end of the file. So a reader has to seek near the end and scan backwards looking for the four-byte signature, potentially across 65,557 bytes.

Searching for a magic number is not the same as knowing where a structure is. If those four bytes happen to appear inside the comment, or inside compressed data near the end of the file, a naive parser can lock onto the wrong one. Different implementations pick different candidates when there’s more than one. This is a recurring source of “this archive opens in one tool and not another.”

You Can Put Anything in Front of a ZIP

If a reader finds the archive by scanning backwards from the end, then whatever sits at the front of the file is not the reader’s problem.

Take a valid 69-byte PNG, take the 241-byte ZIP, and concatenate them with cat. No special tooling:

$ file polyglot.png
polyglot.png: PNG image data, 1 x 1, 8-bit/color RGB, non-interlaced

$ unzip -l polyglot.png
  Length      Date    Time    Name
---------  ---------- -----   ----
       12  08-10-2026 16:44   hello.txt
       13  08-10-2026 16:44   second.txt
---------                     -------
       25                     2 files

One 310-byte file. An image viewer reads the PNG signature at byte 0 and renders an image. An archive tool scans backwards, finds the EOCD, and extracts two files. Both are correct. Neither is being fooled by a trick; they’re each doing exactly what their format says to do.

This is the mechanism behind self-extracting archives, where the front of the file is a real executable and the back is a real ZIP. The same property is why “GIFAR” attacks worked: a file that a server accepted as a harmless image was loaded by Java as an archive of classes.

It also means the offsets inside the Central Directory are relative to the start of the archive, not the start of the file, and readers have to work out that difference. Prepending data shifts everything, and well-behaved parsers cope by computing the delta between where the EOCD says the directory should be and where it found it.

Two Indexes, One File

Back to that detail from earlier: every file’s name and metadata are stored twice, in the Local File Header and again in the Central Directory.

Nothing enforces that they agree.

Here’s the same archive with only the Central Directory copy of the first filename patched from hello.txt to BOGUS.txt. The local header is untouched:

$ unzip -l mismatch.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       12  08-10-2026 16:44   BOGUS.txt
       13  08-10-2026 16:44   second.txt

  local file header at offset 0 still says: hello.txt

The archive is not corrupt. unzip lists it happily. It just contains two different answers to “what is this file called,” and which one you get depends on which structure your parser decided to trust.

Now imagine two programs reading the same archive, one checking a signature and the other extracting files. That’s the Android “Master Key” bug from 2013, and the detail is better than the summary.

An APK is a ZIP. The attacker puts two entries in it, both named classes.dex. Android’s Java verifier loaded entries into a map keyed by filename, so a duplicate name overwrote the earlier one and the last entry was the one whose signature got checked. The native installer used a hash table with linear probing that didn’t replace on collision, so the first entry was the one that got loaded and run. Plant malicious code first, legitimately signed code second, and the device verifies one file and executes the other.

A second bug the same year came from the same “two readings, one file” family, via a signed integer. The extra-field length is a 16-bit value, and the Java code read it signed. A length of 65,533 (0xFFFD) sign-extends to −3. Since the offset of the compressed data is computed by adding that length, a negative value moves the read pointer backward into the header region instead of forward past it.

The lesson generalizes past ZIP. Any format that stores the same fact twice has to decide what happens when the copies disagree, and “the spec doesn’t say” is the same answer as “attackers decide.”

Offsets Are Just Numbers

The Central Directory locates each file by offset. Nothing in the format says two entries can’t point at the same bytes.

The classic zip bomb didn’t need that. 42.zip is 42 kilobytes of archives nested five layers deep, sixteen at each layer, unpacking to roughly 4.5 petabytes. The defense is obvious once you’ve seen it: cap recursion depth, don’t auto-extract nested archives.

David Fifield’s 2019 construction doesn’t recurse at all. It expands in a single pass, so depth limits are irrelevant. The trick is overlap: many Central Directory entries reference one shared kernel of compressed data, and each entry’s compressed stream uses DEFLATE’s stored-block mode to quote the next entry’s local file header as literal bytes. Entries nest inside each other, and output grows quadratically against input.

He published several, and they aren’t interchangeable:

File	Compressed	Uncompressed	Ratio	Needs Zip64
`zbsm.zip`	42 KB	5.5 GB	~130,000:1	No
`zblg.zip`	10 MB	281.4 TB	~28,000,000:1	No
`zbxl.zip`	46 MB	4.5 PB	~98,000,000:1	Yes

The Zip64 requirement on the largest one matters, because not every reader supports Zip64, which makes the merely-enormous version the more portable weapon.

None of these are malformed files. Every one is a valid archive that a conforming parser is supposed to accept. The format allows two entries to describe the same bytes, and no rule anywhere says the total uncompressed size has to bear any relationship to the file you’re holding.

Everything Is Secretly a ZIP

Once you know the structure, you start recognizing it:

.docx, .xlsx, .pptx are ZIP archives of XML
.jar, .war, .apk are ZIP archives of class files and resources
.epub is a ZIP of XHTML
.odt, .ods are ZIP of XML again

That’s not a coincidence or a hack. ISO/IEC 21320-1, “Document Container File,” defines a constrained ZIP profile for exactly this use. It narrows the format so a .docx reader doesn’t have to implement all of ZIP’s accumulated history: compression must be stored or deflated and nothing else, and the various encryption and digital-signature mechanisms in the original spec are all forbidden.

It’s a narrowing, not a rewrite. Zip64 version 1 is still permitted, for instance; only version 2 is ruled out. The profile is best understood as a list of the parts of ZIP that turned out to be a bad idea.

Which means the next post in this series is mostly about a ZIP file with XML inside it. You already know half of how a Word document works.

Sources

PKWARE APPNOTE.TXT — the original and still-authoritative ZIP specification, currently version 6.3.10; §4.3.16 defines the end of central directory record
ISO/IEC 21320-1:2015 — the constrained ZIP profile used by document formats. Fair warning, this one is a paid ISO standard; the catalog page tells you what it covers but you cannot read the text without buying it
Library of Congress format description for ZIP — history and preservation notes
David Fifield: A Better Zip Bomb — the overlapping-stream construction

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Aug 17, 2026 Programming security File-formats Compression Zip

pnpm 11 Made the Safe Thing the Default
Protecting against supply chain attacks requires vigilance. You have to audit your dependencies. You have to pin your versions. You have to review your install scripts. All of these are great things to do, but they require sustained effort.

pnpm 11 took the obvious thing and made it the default. They changed the waiting period. minimumReleaseAge defines the minimum number of minutes that must pass after a version is published before pnpm will install it.

Before version 11 the default was 0. In version 11 the default is 1440 minutes, which is 24 hours, and it applies to everything.

Most malicious packages get discovered and pulled from the registry within minutes, or at most an hour. So what a day of patience buys you is that you’ve eliminated the potential for dependencies sneaking in that haven’t been fully vetted.

npm already followed suit, and so did everyone else. This is now table stakes across the ecosystem:
- pnpm got there first with minimumReleaseAge, measured in minutes, back in 10.16 in September 2025.
- Yarn shipped npmMinimalAgeGate, also minutes, in 4.10.0 that same month.
- Bun added minimumReleaseAge in 1.3 in October 2025, measured in seconds, plus a minimumReleaseAgeExcludes list for packages you trust.
- npm landed min-release-age in 11.10.0 in February 2026, measured in days.
I guess they couldn’t agree on the unit of time for their release age setting.

As far as I know, in all of them besides pnpm, the cooldown is opt-in. It’s not the default. So you have to know the setting exists and you have to go turn it on.

Here are some other things that changed in pnpm 11:
- allowBuilds replaces onlyBuiltDependencies, which was removed in v11. It’s a map of which packages may run build scripts. Anything not listed is disallowed and treated as unreviewed.
- strictDepBuilds defaults to true. Installation exits with a non-zero code if any dependency has unreviewed build scripts, so this fails your CI rather than printing a warning nobody reads.
- verifyDepsBeforeRun defaults to install. Before pnpm run or pnpm exec, it checks whether your dependency state matches the lockfile. Other options are warn, error, prompt, and false.
- dangerouslyAllowAllBuilds defaults to false, and the name is doing exactly the work it should. Setting it true lets every dependency, transitive ones included, run install scripts now and in the future.
The clear pattern here is that an automated or unintentional action should be blocked, not permitted with a warning.

Sure, there is somewhat of a cost here. The delay means you can’t immediately install a new version that was just published unless you flip the flag. I can see the allowBuilds migration being somewhat of a hassle, because the first time you install after upgrading you’re going to get a list of packages that want to run build scripts. It’s easy to be lazy and approve all of them without thinking.

With the tools we have available to us these days, we can ask an agent to review the build scripts. This is the right thing to do. Find a way to pin the dependency to fix transitive version issues. The inner engineer in all of us needs to understand why build scripts are dangerous, and what to be careful of, so that you can ask your subagent to go and see if that’s a problem, or if that problem has been fixed with the new version. It’s up to the human in the loop to ensure that the agents are doing their due diligence.

When building software, we should also look for other paths to optimize, and make the lazy path the safest one, because chances are that’s going to become the default.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 12, 2026 security javascript Tooling Supply-chain Nodejs

Enjoyed this?
gadievron/raptor: Raptor turns Claude Code into a general-purpose AI offensive/defensive security agent. By using Claude.md and creating rules, sub-agents, and skills, and orchestrating security tool usage, we configure the agent for adversarial thinking, and perform research or attack/defense operations.

Raptor turns Claude Code into a general-purpose AI offensive/defensive security agent. By using Claude.md and creating rules, sub-agents, and skills, and orchestrating security tool usage, we confi…

Jul 4, 2026 Claude links agent automation code security vulnerability
Your AI Coding Agent Can Read Every Secret on Your Machine
Every developer running an AI coding agent has handed that agent the keys to their machine. Not metaphorically. Literally. The agent runs as your user. It can read every file you can read, execute every command you can execute, and hit every API your stored credentials authorize.

For most workflows, that’s the point. You want the agent to read your code, modify your project, ship your work. But there’s a quieter implication: the agent can also read your .env files. It can invoke your secret-management tooling. It can grep for API_KEY= across your home directory. And nothing in the agent stack says “wait, you didn’t ask for this.”

Same-UID isolation isn’t isolation. It’s the absence of isolation labeled politely.

The usual answer to “keep secrets safe from your coding agent” is: don’t store them where the agent can find them. Use a cloud secret manager. Rotate aggressively. These are good practices, and for local development, they’re often impractical. The agent is going to encounter secrets whether or not your security-best-practices doc approves.

So over the last week, I built an audit subsystem into lsm, my Local Secrets Manager. The whole thing is designed to answer one forensic question: did anything weird touch my secrets last night?

The Threat Model

A defense without a threat model is theater, so let me be specific.

The threat isn’t a sophisticated remote attacker. lsm is public, open-source code. The threat isn’t a buggy lsm either; bugs happen, and the user can read the source.

The threat is the agent layer running adjacent to lsm. Coding agents have legitimate access to a wide swath of your filesystem. They’re imperfect at intent inference. They sometimes get prompt-injected. They sometimes run in the background while you’re asleep. When an agent calls lsm get prod DATABASE_URL, the action is indistinguishable from you doing the same thing. The audit log’s job is to make those calls retrospectively distinguishable.

A secondary threat is an agent covering its tracks. If something reads a secret and then edits the audit log to erase the evidence, the log is worse than useless.

What Got Built

The audit subsystem records every access as a structured event: a sequence number, a timestamp, the action, the app and environment, an Actor block describing the calling process, and two cryptographic fields linking each event to the previous one.

The Actor block was the interesting design problem. It captures parent process ID, parent process name, TTY device path (or empty if there’s no terminal), current working directory, an agent marker derived from environment variables that tools like Claude Code, Cursor, Aider, and Continue set, and the calling user ID. Every field is captured every time. No omitempty. UID zero is a real, meaningful value, and silently dropping it would be a footgun.

Events land in a hash-chained JSONL file at ~/.lsm/audit.jsonl. Each row carries the SHA-256 of the previous row plus its own body. If anyone edits, inserts, or deletes a row in the middle, the next row’s prev no longer matches and lsm audit verify surfaces the break.

The chain doesn’t catch tail truncation. If you chop off the end of the file, what’s left is internally consistent. A sidecar file storing the last expected hash is the obvious fix, and I deliberately rejected it. lsm is public code. Any local attacker who knows about the sidecar can rewrite both files in lockstep. Tail-truncation detection is deferred to the off-machine path: when events ship to a remote stack, the last hash naturally lives somewhere the local attacker doesn’t control.

Reading the Log

Three commands cover the read side. lsm audit tail does what you’d expect. lsm audit show <seq> prints a single event. lsm audit query is the workhorse, with every field as a filterable dimension: --app, --env, --event, --parent-comm, --agent-marker, --tty present|absent, --since, --until. Output is JSONL when piped and columnar text when interactive.

Then there’s lsm audit suspicious, which runs four hard-coded detectors in one pass:
- Outside hours. Events whose timestamps fall outside 07:00–23:00. The 3 a.m. canary.
- Burst. More than N events from a single parent process within a sliding window. The runaway-agent canary.
- New parent_comm. Process names not seen in the prior 30 days. The “what is this new thing” canary.
- Non-interactive, no agent. No TTY, no recognized agent marker. The “what is even running this” canary.
A single event can stack reasons. A 3 a.m. burst from an unknown parent is unambiguously interesting.

The detector doesn’t learn baselines, doesn’t call out to an ML model, doesn’t require a service. High-signal patterns are obvious patterns, and obvious patterns are well-served by hard-coded predicates.

Shipping Events Off the Box

If you already run an observability stack, lsm can ship audit events over OTLP (the OpenTelemetry wire protocol). Three design choices matter here.

The local file sink is always authoritative. The remote sink is a mirror, not a replacement. An lsm operation never fails because the remote endpoint is down.

Redaction is allowlist-based. App and environment names are HMAC-hashed with a per-host salt before becoming labels. The TTY device path is dropped and replaced with a tty_present: true/false boolean. Secret values, cwd, hash, prev, and the schema version never leave the host. Secret names are replaced with key_present: true markers; the remote observer can see that a key was accessed, never which key.

Events whose name starts with audit. (chain failures, suspicious matches, sink drops) are always local. Telling a remote attacker that local integrity has been compromised is counterproductive.

What’s Still Open

The most important non-feature: no command in lsm emits events yet. set doesn’t log. get doesn’t log. delete doesn’t log. The plumbing is complete, the calls are not wired in. Each emit site needs careful thought about which fields are appropriate, whether the event should be local-only, and how it interacts with sensitive operations. That’s the next chunk of work.

The agent-coding era is normalizing a model where AI tools have wide-ranging access to developer machines. The premise that the agent operates as a fully-trusted local user is unlikely to change soon. Managing the risk means visibility. It means being able to answer “what touched my secrets last night” with a record the agent couldn’t silently rewrite.

The code is at github.com/llbbl/lsm. The full design lives in docs/observability.md.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
May 18, 2026 DevOps AI Programming security

Enjoyed this?

Buying Supply Chain Security in 2026: A Vendor Map

The last post was for solo developers and people without a security budget. This one is for everyone else: the platform engineers, the security leads, and the directors who are getting pitched by four different supply chain security vendors a week and need to figure out which, if any, of them are worth signing a contract with.

The honest answer is that the vendor landscape in 2026 is overheated. Every SCA company is now also a malicious-package firewall company. Every malicious-package firewall company is also pitching AI-native remediation. The pricing pages are mostly “Contact Sales.” And underneath all of it, the actual problem these tools solve splits cleanly into three layers, and you should know which layer you’re buying.

The Three Layers

Layer 1: Update automation. Dependabot (free, GitHub-native) and Renovate (free, more configurable) generate pull requests when new versions of your dependencies are released. They don’t find vulnerabilities. They just shrink the window where you’re running outdated code. Dependabot is the right answer for most teams under 50 engineers. Renovate is what you reach for when you’re tired of triaging 80 individual PRs a week and want grouped updates with auto-merge based on community confidence signals. Neither costs anything. Both should be on.

Layer 2: Software Composition Analysis (SCA). Parses your lockfiles, matches dependencies against CVE databases, tells you what’s vulnerable. The open-source side of this is fully mature: Trivy, Grype, OWASP Dependency-Check, and OWASP Dependency-Track collectively cover most of what you’d pay Snyk for ten years ago. Dependency-Track in particular is a serious tool. It ingests CycloneDX and SPDX SBOMs, tracks portfolio-wide risk, and integrates EPSS scoring. If you self-host it, the bill is zero.

The thing the commercial vendors actually sell at this layer is reachability analysis. A vulnerability in a transitive dependency that you import but never actually call is technically a CVE in your inventory. Realistically it’s noise. Snyk, Endor Labs, and Mend.io all build call-graph analysis that determines whether a vulnerable code path is actually invoked by your application. Endor Labs claims their reachability reduces actionable alerts by 90 to 95%. That number is marketing, but the underlying capability is real, and it’s the single biggest differentiator between commercial SCA and the open-source stack.

Layer 3: Malicious package firewalls. This is the layer that didn’t exist five years ago. Tools like Socket, Phylum, Endor Labs, and Sonatype Repository Firewall sit between your developers and the public registries and analyze package behavior before installation. Socket evaluates 70+ behavioral indicators: does the package read OAuth tokens from disk, does it use marshal.loads to self-deobfuscate, does it inject into HTTP headers. This is the only layer that defends against zero-day malicious packages, because SCA fundamentally can’t. There’s no CVE for “this package was uploaded ten minutes ago and steals AWS keys.”

What This Actually Costs

The pricing pages tell you most of what you need to know about who each vendor is for.

Vendor	Pricing	Who it’s for
Dependabot	Free	Everyone on GitHub
Socket	Free up to 1000 scans/mo, Team $25/dev/mo, Business $50/dev/mo	Developers who want low-friction zero-day protection
Snyk	Free tier (100-300 tests/mo per product), Team $25/dev/mo (5-10 dev cap), Ignite ~$105/dev/mo, Enterprise custom	Teams that want SCA + SAST + IDE integration in one bundle
Endor Labs	Custom (free tier for small OSS teams)	Orgs drowning in CVE noise; multi-language including C/C++ and Rust
Mend.io	$300-$1000/dev/year	Enterprise environments that want consolidated dashboards
Sonatype	$6K-$150K+ in bundled tiers	Large regulated enterprises that need a centralized artifact gateway
Phylum	Custom enterprise	Teams that want programmatic policy via Open Policy Agent

Two patterns stand out. Socket and Snyk are product-led growth plays with transparent per-developer pricing, predictable as you scale, accessible at the lower end. Sonatype, Mend.io, and Phylum are enterprise sales motions with significant minimums and multi-month implementation cycles. Endor Labs sits awkwardly in the middle (mid-market and enterprise deals) with credible reachability claims that are hard to replicate with open source.

The Real Cost of “Free”

The argument for going all-in on open source, Dependabot plus Trivy plus Dependency-Track plus maybe Socket’s free tier, looks compelling on the spreadsheet. The honest math is more complicated.

Running this stack at a 100-engineer organization requires somebody to maintain the Dependency-Track server, tune the rulesets to keep false positives from drowning your security team, manually triage alerts that have no reachability context, and respond to the inevitable “is this critical CVE actually exploitable in our environment?” questions from leadership. Realistic estimates put that workload around 20 to 30 hours per week — call it half an FTE of senior engineering time, which fully-loaded lands in the low six figures per year. That’s not zero, and it’s the line item that “we’ll just use open source” plans consistently leave out of the spreadsheet.

The flip side is the Endor Labs ROI pitch: 90% noise reduction means 9 fewer FTEs needed for triage in a 300-dev org, which they price at roughly $1.5M in saved salary against a five-figure license. That’s a vendor calculation, so take it with the appropriate salt. But the underlying logic that alert noise has real labor cost is correct, and it’s the part most “we’ll just use open source” plans underestimate.

For a team of 5 to 50 engineers: Dependabot or Renovate on, Socket’s free tier or paid Team plan for firewall coverage, and npm audit / pip-audit / cargo-audit running in CI. Total spend: $0 to roughly $1,500/month at the high end. This is the configuration that covers 80% of the threat for a small fraction of what a Snyk or Mend contract costs.

For 50 to 300 engineers: the math starts favoring a paid SCA platform with reachability. Snyk if you also want SAST in the same tool. Endor Labs if you have a polyglot codebase (especially anything with C++ or Rust) and severe alert fatigue. Keep Socket or Phylum as a separate firewall layer. The firewall vendors are still meaningfully better at malicious-package detection than the SCA vendors who bolted it on.

For 300+ engineers in a regulated industry: you probably need Sonatype or JFrog as a centralized proxy whether you want them or not, because compliance demands a single audited path from developer to registry. Bundle it with Endor Labs or Mend for the reachability layer.

What I would not do is buy the platform pitch, the “one tool for SCA + SAST + secrets + container scanning + firewall + AI remediation.” Those bundles exist because the vendors want a bigger contract, not because the unified product is actually best-of-breed at any single thing. The companies winning each individual layer (Socket for firewalls, Endor Labs for reachability, Trivy for open-source SCA) are doing so by being focused.

Closing the Series

Four posts in: the threat model, the per-ecosystem mitigations, local isolation for the budget-constrained, and now the commercial landscape for everyone else. The unifying thesis across all of them is that supply chain security is not solved by a single tool or a single layer. It’s a stack. Lockfiles at the bottom, audit tooling above that, behavioral analysis on top, isolation as the last line of defense. The right composition depends on who you are and how much risk you can afford to absorb. If your stack right now is “we trust the registry,” you are the threat model.

Sources

Supply Chain Security Tool Selection Framework - SoftwareSeni
Endor Labs vs Snyk: SCA, SAST, and Containers Compared
Malware Package Firewall: Block Threats Before They Hit Your Code
Socket Pricing
Introducing Socket Firewall
Snyk Software Pricing & Plans 2026 - Vendr
Endor Labs Pricing
Mend.io Pricing
Sonatype Nexus Pricing Guide 2026 - CloudRepo
Open Source vs Commercial SCA Tools Comparison - Safeguard
OWASP Dependency-Track

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

May 17, 2026 DevOps security Tooling Supply-chain

Sandboxing AI Agents Without Buying Anything
The previous post (and the one before it) covered the threat model and the per-ecosystem mitigations: lockfiles, --ignore-scripts, cargo-audit, Trusted Publishing. All of that helps. None of it answers the question that keeps me up at night, which is: what happens when an AI agent on my laptop installs a malicious package, and the malicious package was the literal point of the operation?

This is the new shape of the threat. You’re not getting compromised because you typed npm install wrong. You’re getting compromised because Claude or Cursor confidently invented a package name that didn’t exist, an attacker registered it five hours ago, and the agent ran pip install hallucinated-thing on your behalf without asking. The agent has shell access. Your SSH keys are right there. Your ~/.aws/credentials file is right there. The entire premise of giving an AI agent the ability to just figure it out depends on it being able to execute untrusted code at the speed of conversation, which is also the worst possible threat model.

If you’re a solo developer, an open-source maintainer, or a startup with no budget for Socket or Endor Labs licenses (more on those next post), the answer isn’t a commercial firewall. The answer is local isolation, and the tools have gotten dramatically better in the last 18 months.

Containers as the Baseline

The minimum viable isolation in 2026 is don’t run untrusted code as your user on your host OS. The cleanest way to do that on macOS or Linux is a devcontainer, a fully described, reproducible Linux environment that VS Code, Cursor, and the Claude Code CLI all natively support. You give the agent the container as its sandbox. Project files mount in. SSH keys, AWS credentials, and the rest of your home directory don’t.

The container runtime matters. Docker Desktop on macOS is a memory pig, 3 to 4 GB resident at idle, with sluggish startup times that make iterative work miserable. OrbStack is the obvious replacement: free for personal use, native Apple Silicon, dynamically allocates memory instead of reserving fixed blocks, and benchmarks show container startup times around 0.2 seconds versus Docker Desktop’s multi-second cold starts. If Docker Desktop is eating half your RAM before you even start Claude Code, OrbStack will give you that memory back.

The thing to internalize, though, is that a container is not a security boundary by default. It’s a deployment mechanism that happens to have isolation properties when configured correctly. Misconfigured developer containers have been implicated in some of the largest crypto-industry breaches of the last few years. The pattern: a container running with privileged flags, or mounting the wrong host directory, turns into a path straight to the host. Containers help. They don’t save you from yourself.

The configuration mistakes that void the isolation:
- Mounting ~/.ssh into the container so the agent can git push. Now any process inside the container can read your SSH keys.
- Mounting your entire home directory as a convenience. Now everything is accessible.
- Running with --privileged or sharing the host’s Docker socket. Container escape becomes trivial.
- Letting the agent run sudo inside the container. The container’s root can chain to host kernel exploits.
Least privilege, applied seriously. The agent gets the project directory and nothing else. If it needs to commit, it pushes through a credential helper that lives on the host, not by mounting your SSH keys.

Lighter-Weight Sandboxes

Spinning up a full container for every test this snippet the LLM wrote interaction is too heavy. There’s a middle layer worth knowing about.

Python. Pyodide compiles CPython to WebAssembly, which means Python code runs in a deny-by-default memory sandbox with no filesystem or network access unless you explicitly grant it. Works great for evaluating LLM-generated snippets, struggles with C extensions and heavy dependencies. safe-py-runner is the pragmatic alternative: it runs Python in a restricted subprocess with timeouts, memory limits, and I/O marshaling. No container needed. For code that absolutely cannot touch your machine, remote V8-isolate services like Deno Sandbox boot pre-snapshotted Python environments in the cloud and air-gap execution entirely.

Rust. The build.rs problem from the last post has no first-class solution yet, but on Linux you can wrap cargo build in Landlock, a kernel feature available on 5.13+ that lets unprivileged processes restrict their own filesystem access. Combined with seccomp-bpf for syscall filtering and cgroups v2 for resource limits, you can run a build script that genuinely cannot read your SSH keys or open arbitrary network sockets. Projects like sandbox-rs wrap these primitives into something usable without writing your own seccomp filters. None of this works on macOS without a Linux VM in the way, which is another reason OrbStack plus a devcontainer is the path of least resistance for most people.

The Mindset Shift

The honest version of all of this: if you’re running AI agents locally, you have to assume they will eventually install something malicious. Not might. Will. The question is whether the blast radius is the contents of one project directory inside a container, or every credential on your machine plus your entire git history. That gap is what isolation buys you.

Containers, Landlock, WASM sandboxes, none of these are particularly hard to set up. They’re just things most developers haven’t bothered with because the threat model didn’t feel real. After Shai-Hulud, faster_log, and a year of watching AI agents pip install whatever they invent, the threat model is real.

Next post I’ll wrap up the series with the commercial side: Socket, Snyk, Endor Labs, Mend, Sonatype, the pricing comparison, and the actual ROI math for whether any of it makes sense for teams below 50 developers.

Sources
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
May 16, 2026 DevOps AI security Supply-chain Containers

Enjoyed this?

security

The Header Is Legible

Two Libraries, Two Answers

The Single-File Database Is Three Files

What To Do About It

Sources

Read It From the Bottom

The Black Box Is a Rect

Doing It Properly and Still Losing

Hard to Get Right

What To Do About It

Sources

Two Kinds of Correct

The Entity System

The Bomb That Got Fixed

What the Rest of It Bought

What To Do About It

Sources

The Norway Problem

It Was Fixed in 2009

Everything Else That Isn’t a String

Two Ways to Weaponize the Convenience

What To Do About It

Sources

The Index Lives at the End

Why Would You Do This?

The Backwards Scan Is Fuzzier Than It Sounds

You Can Put Anything in Front of a ZIP

Two Indexes, One File

Offsets Are Just Numbers

Everything Is Secretly a ZIP

Sources

The Threat Model

What Got Built

Reading the Log

Shipping Events Off the Box

What’s Still Open

The Three Layers

What This Actually Costs

The Real Cost of “Free”

What I’d Actually Recommend

Closing the Series

Sources

Containers as the Baseline

Lighter-Weight Sandboxes

The Mindset Shift

Sources