Parsing

Every XML Feature Is One You Turn Off
Last post ended on a question. XML is the format in this series that specified everything, so what did all that specifying buy?

An answer for every question, and an attack surface made entirely of answers.

The other formats in this series failed by leaving things out. CSV never had a standard. JSON declined to say what a number means. YAML wrote the rules down and then most implementations kept using the old ones. XML failed the other way. It defined a schema language, a query language, a transformation language, a namespace system, and an entity system, and the entity system will read files off your disk.

Two Kinds of Correct

Start with something XML got right, because it is the only format here that made this distinction at all.

The specification defines two separate bars:

A data object is an XML document if it is well-formed, as defined in this specification. In addition, the XML document is valid if it meets certain further constraints.

Well-formed is syntax. Tags match, there is one root, attributes are quoted. Valid is semantics: the document declares a schema and conforms to it.

CSV has neither concept. JSON has only the first one. XML separated them in 1998, which was ahead of its time, and then almost nobody used the second one. Most XML in production is well-formed and unvalidated, which means it has exactly the same guarantees as JSON with more punctuation.

The Entity System

XML documents are built from entities. Five are predefined, and you know them:
```
&amp;  &lt;  &gt;  &quot;  &apos;
```
You can also declare your own in the document type declaration, which is how you get constants in a config file. And an entity can be declared to pull its content from somewhere else:
```
<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "file:///tmp/xmltest/secret.txt">
]>
<foo>&xxe;</foo>
```
That is a legal XML document. SYSTEM means “go get it.” Here is what a parser does with it:
```
ElementTree  ParseError: undefined entity &xxe;
lxml         XMLSyntaxError: Entity 'xxe' not defined
defusedxml   EntitiesForbidden(name='xxe', system_id='file:///tmp/xmltest/secret.txt')
lxml, resolve_entities=True    'SECRET-CANARY-12345\n'
```
The first three refuse. The fourth read a file off the disk and put its contents in the document tree.

The difference between the second line and the fourth is one keyword argument. Not a patch, not an old version, not a misconfiguration. A parser option, on a current library, named after something that sounds like ordinary XML processing. Of course you want entities resolved. Entities are how the format works.

This is XXE, and it is not a bug in any implementation. Every one of those parsers is behaving as specified. The specification says an external entity is fetched, so the ones that fetch it are correct and the ones that refuse are deliberately non-conforming for your safety.

The Bomb That Got Fixed

The same entity system nests, which produces the XML version of the billion laughs attack:
```
<!DOCTYPE lolz [
  <!ENTITY lol "lol">
  <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
  <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
]>
```
Each level multiplies by ten. A 291-byte file at three levels expands to 3,000 characters. Add two more levels and the arithmetic says 300,000, and the file is still under 500 bytes.

Except it doesn’t, and this is the part worth reporting honestly. At five levels, current libxml2 stops:
```
XMLSyntaxError: Maximum entity amplification factor exceeded,
see xmlCtxtSetMaxAmplification
```
It refuses even with resolve_entities=True. Somebody went and put a ratio limit in the parser, and it works. Compare that to YAML, where safe_load still expands aliases without complaint, and the equivalent 202-byte payload produced 74,732 nodes with no objection at all.

So the XML ecosystem fixed the denial-of-service and left the file-read one flag away. That ordering tells you something about which failure people actually hit.

What the Rest of It Bought

XML shipped an enormous amount of specification, and the pieces are individually good:
- XSD defines 19 primitive datatypes with real inheritance, so <price>12.50</price> can be a decimal rather than a hopeful string. This is precisely what CSV lacks and what JSON refuses to commit to.
- XPath addresses any node in a document without writing a traversal.
- XSLT transforms one document into another declaratively.
- Namespaces let two vocabularies coexist in one file without colliding, which is the problem every format in this series either ignores or solves with a naming convention.
None of that is bad engineering. Read the list again and notice that it is a description of the problems the other five posts were about. XML solved them. In 1998.

The cost was that using XML correctly means knowing which parts to switch off, and the defaults were set in an era that assumed documents came from people you knew. Every hardening guide for XML is a list of features to disable: no DTDs, no external entities, no network access, no schema fetching.

There is a version of this where the lesson is “XML was too complicated.” I don’t think that’s it. The formats that replaced it are simpler and they have the same problems, plus the ones XML had already solved. Nobody misses XSLT, and everybody has now written their own worse version of it.

What To Do About It
- Disable DTD processing entirely unless you know you need it. In Python that is defusedxml; in Java it is disallow-doctype-decl. This closes XXE and entity expansion in one move.
- Never enable resolve_entities on input you did not write. It is the single flag that turns a parser into a file reader.
- Validate against a schema at the trust boundary, not just parse. Well-formed is not a security property. It is barely a correctness property.
- Stream large documents. SAX and StAX read in constant memory; DOM builds the whole tree first, which is its own denial of service if the document is attacker-sized.
- Know your parser’s defaults and pin them explicitly. They have changed over time, usually toward safety, and code that relies on a safe default is one dependency upgrade from a different one.
XML answered every question these posts have raised. It has types, a validation model, a query language, and a namespace system, and it had all of them a decade before the formats that replaced it. What it could not do was make the safe path the default one, and that turned out to matter more than any of the rest of it.

That is the actual pattern across this whole series. Not that formats are underspecified or overspecified, but that the defaults are the specification most people ever use.

Sources
- XML 1.0 (Fifth Edition) — 26 November 2008; the well-formed and valid definitions, and the five predefined entities
- XML 1.0 (First Edition) — 10 February 1998, the original Recommendation
- XML 1.1 — 4 February 2004; the revision almost nobody adopted
- XSD 1.1 Part 2: Datatypes — the 19 primitive datatypes
- OWASP XXE Prevention Cheat Sheet — per-parser hardening settings
- defusedxml — the Python library that turns the dangerous parts off for you
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 22, 2026 Programming security File-formats Parsing Xml

Enjoyed this?
Norway Is Not a Boolean
JSON’s problem is that its specification is too small. It tells you 9007199254740993 is a well-formed number and then declines to say which number.

YAML went the other way. The 1.2.2 specification is a book. It has a formal grammar, a chapter on recommended schemas, and an answer for nearly everything. And it will still read your config file and decide, without asking, that Norway is false.

The Norway Problem

Here is a config file. Every value in it is a string that a human would read as a string.
```
country: NO
duration: 1:20
mode: 0755
```
Three parsers, on those exact bytes:
```
PyYAML 6.0.3   {'country': False, 'duration': 80,     'mode': 493}
ruamel.yaml    {'country': 'NO',  'duration': '1:20', 'mode': 755}
js-yaml        {"country":"NO",   "duration":"1:20",  "mode":755}
```
NO is the ISO 3166 code for Norway. PyYAML returns the boolean false, because YAML 1.1 recognized twenty-two spellings of true and false, and NO is one of them. The spec lists them as a single regular expression:
```
y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF
```
Count them. Twenty-two. Six of those are country codes, single letters, or ordinary English words that appear in real data. Nothing in the file said “this is a boolean.” The parser inferred it from the shape of the text, and the shape of the text was two letters.

1:20 became 80 because YAML 1.1 supported sexagesimal integers, so a duration is read as base 60. One times sixty, plus twenty.

0755 became 493 because a leading zero meant octal. That is a file mode that no longer means what it says.

It Was Fixed in 2009

This is the part that makes YAML different from the other formats in this series.

CSV never had a standard. Markdown had too many. YAML had exactly one problem, everybody agreed it was a problem, and the working group fixed it. YAML 1.2 arrived in 2009 and threw all of it out. Base 60 is gone. Implicit octal is gone. The Core schema recognizes true and false and their case variants, and nothing else.

Seventeen years later, the two YAML 1.2 parsers above return strings, and PyYAML returns False.

PyYAML implements YAML 1.1. It is the default YAML library for Python, it is what pip install pyyaml gives you, and the specification it implements was superseded when the iPhone 3GS was current. The fix exists. It shipped. Most of the ecosystem simply stayed where it was, because changing the type of NO in a minor release breaks every config file that relied on it.

A format can be fixed and still be broken, if the fix arrives after the implementations do.

Everything Else That Isn’t a String

The country-code case is famous. It is not the only one, and the rest are quieter:
```
version: 1.10   ->  1.1        (float, and .10 became .1)
build: 010      ->  8          (octal)
port: 8080      ->  8080       (int, fine, until you concatenate it)
answers: [y, n] ->  ['y', 'n'] (strings)
answers: [yes, no] -> [True, False]
```
The first one is the one that should bother you. A semantic version of 1.10 parses as the float 1.1, which is a different version, and it does it silently in a file whose entire job is to record which version you meant.

And note the last two lines. y and n stay strings in PyYAML while yes and no become booleans, because PyYAML’s resolver implements a narrower set than the 1.1 spec’s regexp advertises. So the answer to “does this parser coerce single letters” is neither yes nor no. It is “some of them, and you have to test.”

Two Ways to Weaponize the Convenience

YAML has anchors. You define a node once with &name and reference it with *name. It is a useful feature for config files with repeated blocks, and it composes.

That is the problem. It composes exponentially.
```
a: &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
```
That file is 202 bytes. Expanding it produces 74,732 nodes, of which 59,049 are copies of the string lol. Add one more line and multiply by nine. This is the billion laughs attack, and the important detail is that safe_load does not stop it. Aliases are not a dangerous tag, they are a core language feature working as designed.

The second way is tags. YAML can annotate a node with a type, and PyYAML historically honored tags that construct arbitrary Python objects:
```
!!python/object/apply:os.system
args: ['id']
```
yaml.load() on untrusted input would run that. It became CVE-2017-18342, CVSS 9.8, published June 2018, with a description that is unusually blunt for the genre: “In PyYAML before 5.1, the yaml.load() API could execute arbitrary code if used with untrusted data.”

The fix took two releases and three years. PyYAML 5.1 deprecated the unsafe default in March 2019. PyYAML 6.0 finally made the Loader argument mandatory in October 2021, so the dangerous call stopped being the short one:
```
>>> yaml.load('a: 1')
TypeError: load() missing 1 required positional argument: 'Loader'
```
The vulnerability was published in 2018. Making the unsafe call harder to type than the safe one landed in 2021.

What To Do About It
- Quote anything that isn’t obviously a number. Country codes, versions, file modes, git SHAs, anything a human would call an identifier. Quoting is never wrong.
- Know which YAML version your parser speaks. If it is Python, assume 1.1 and the Norway problem unless you chose otherwise. ruamel.yaml gives you 1.2.
- Never call yaml.load on input you did not write. safe_load, always. On PyYAML 6 the language makes you say which you meant, which is the correct design.
- Bound the input. safe_load is not a defense against alias expansion. If you parse YAML you did not author, cap the document size before it reaches the parser.
- Use a schema. The value of a schema here is not validation, it is that it declares the type instead of letting the parser guess it from the characters.
YAML’s failure is the opposite of JSON’s, and it produces the same result. JSON declined to say what values mean, so implementations disagreed. YAML said what values mean in enormous detail, got it wrong in 2005, corrected it in 2009, and the correction never fully landed.

Next in this series is XML, which is the one format here that did specify everything. It has a schema language, a query language, a transformation language, and a namespace system. It is worth asking what all of that bought.

Sources
- YAML 1.2.2 Specification — October 2021; the schemas chapter and the rule that tabs “must not be used in indentation, since different systems treat tabs differently”
- YAML 1.1 Boolean type — the twenty-two-form regexp, working draft dated 2005
- CVE-2017-18342 — the yaml.load() RCE, CVSS 9.8
- PyYAML CHANGES — 5.1 (2019) deprecated the unsafe default, 6.0 (2021) made Loader required
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 21, 2026 Programming security File-formats Parsing Yaml

Enjoyed this?
Your JSON Parser Disagrees With Mine
Markdown has too many specs. CSV has one that nobody agreed to follow. JSON is supposed to be the happy ending: a grammar small enough to print on a business card, standardized twice over, and elevated to a full Internet Standard.

It is, and the parsers still disagree with each other about what your file says.

Not about whether it’s valid. About what the values are.

The Same Number, Three Answers

Here is a JSON document. It is unambiguously valid by every specification.
```
{"id": 9007199254740993}
```
Three parsers, on the same machine (Node 24.13.0, Python 3.13.12, jq 1.8.2), on those exact bytes:
```
node   : {"id":9007199254740992}
python : {"id": 9007199254740993}
jq     : {"id":9007199254740993}
```
Node gave back a different number than the one in the file. It didn’t error, didn’t warn, didn’t round-trip. The last digit changed from 3 to 2.

The reason is that JSON’s grammar allows a number to be any sequence of digits, but JavaScript represents every number as an IEEE 754 double. Above 2⁵³ the integers stop being exactly representable, and Number.MAX_SAFE_INTEGER is 9007199254740991. Our value is two past it.

This is not a JavaScript bug. Node is behaving exactly as specified. The specification simply declines to say how many digits a number may have, and every implementation answers that question with whatever its host language does.

Those version numbers matter, which is its own version of the problem. jq only began preserving decimal literals in 1.7, whose release notes list “use decimal number literals to preserve precision.” Run that same file through jq 1.6 and it goes through a double and hands you Node’s answer. The tool doesn’t just disagree with other parsers. It disagrees with its own past self.

If you have ever wondered why APIs send 64-bit IDs as strings, this is why. Twitter’s snowflake IDs, database primary keys, anything above 2⁵³ has to be quoted or it silently degrades in half the ecosystem.

Duplicate Keys Are Legal
```
{"role": "user", "role": "admin"}
```
RFC 8259 says names within an object should be unique. Should, not must. And it goes on to describe what happens otherwise as varying between implementations.

In practice:
```
node   : {"role":"admin"}
python : {"role": "admin"}
jq     : {"role":"admin"}
```
All three take the last one. That’s the common behavior, and it is not required.

The RFC itself spells out all three possibilities:

When the names within an object are not unique, the behavior of software that receives such an object is unpredictable. Many implementations report the last name/value pair only. Other implementations report an error or fail to parse the object, and some implementations report all of the name/value pairs, including duplicates.

That third option, keeping both, isn’t even representable in most languages’ object types. Nicolas Seriot tested parsers across a dozen languages against cases like this one and concluded there are “no two parsers that agree on what is wrong and what is right.”

Python will show you both if you ask:
```
raw pairs: [('role', 'user'), ('role', 'admin')]
```
The pairs are all there in the document. Choosing one is an interpretation layered on top of parsing.

Now put two parsers in one system. Apache CouchDB did, and it became CVE-2017-12635.

CouchDB used an Erlang parser for authentication and a JavaScript engine for the validation that runs when a document is written. The Erlang parser resolved duplicate keys to the first value. The JavaScript engine resolved them to the last. So a request like this:
```
{"roles": ["_admin"], ..., "roles": []}
```
was read by the write-time validation as an ordinary unprivileged user, because it saw the last roles key and found it empty. It was then read by the authentication layer as an administrator, because that saw the first one. Non-admin users could grant themselves admin.

CouchDB’s fix was to change the Erlang parser to take the last key, matching JavaScript. Not because last-wins is correct, but because agreeing is correct.

This is the JSON version of the ZIP two-index problem from a few posts back. When a format permits two answers to the same question, the gap between two components is where the vulnerability lives.

NaN Is Not JSON, and Python Emits It Anyway

JSON has no way to express not-a-number or infinity. The grammar has no room for them.

Python’s standard library writes them regardless:
```
>>> json.dumps({"a": float("nan"), "b": float("inf")})
'{"a": NaN, "b": Infinity}'
```
That output is not JSON. It’s Python’s default behavior, and it produces a file that other parsers reject or mangle. Feeding those exact bytes onward:
```
node : SyntaxError - Unexpected token 'N', "{"a": NaN, "b": "... is not valid JSON
jq   : {"a":null,"b":1.7976931348623157e+308}
```
Node’s response is correct and useful: this is not JSON, here’s where it broke.

jq’s response is the one that should worry you. It accepted the invalid document and made up values. NaN became null. Infinity became 1.7976931348623157e+308, the largest finite double. No error, no warning. If that ran in the middle of a data pipeline you would get numbers out the other end, and they would be wrong in a way no downstream check is likely to catch.

The same divergence shows up with a merely-enormous exponent, which is valid JSON:
```
input: {"v": 1e999}

node   : {"v":null}
python : {'v': inf}
jq     : {"v":1E+999}
```
Three parsers, one valid input, three different values. Node converts to infinity then serializes it as null because it can’t represent infinity on the way out. Python gives you a float infinity object. jq preserves the literal.

Why a Small Spec Doesn’t Save You

JSON’s specifications are good, and they are small. The problem is that they specify syntax, and almost every failure above is about semantics.

The grammar tells you 9007199254740993 is a well-formed number. It does not tell you what number it is, because that would require committing to a numeric model, and committing to a numeric model would have meant excluding some language from implementing JSON natively. The looseness is why JSON is everywhere. It is also why the same bytes mean different things in different places.

The standards process eventually acknowledged this. RFC 7493 defines I-JSON, a restricted profile that closes these holes: no duplicate names at all, numbers that should not exceed what an IEEE 754 double holds exactly, high-precision values recommended to travel as strings, and mandatory UTF-8. Only the duplicate-name rule is a hard MUST NOT, which tells you something about how much of this was still negotiable in 2015.

I-JSON is what most people think JSON already is. It exists as a separate document precisely because JSON isn’t that.

What To Do About It
- Send large integers as strings. Anything that could exceed 2⁵³: IDs, timestamps in nanoseconds, financial values in minor units.
- Reject duplicate keys at your trust boundary rather than letting your parser pick. Most libraries offer a hook.
- Don’t let a language’s default serializer decide whether it emits valid JSON. Python needs allow_nan=False to be honest.
- Validate before you transform. A parser that repairs invalid input is more dangerous than one that rejects it, because the repair is silent.
- Target I-JSON for anything crossing a system boundary. It costs nothing and removes the whole category.
JSON did not fail. It succeeded so completely that it got implemented hundreds of times by people reading a short document, and a short document leaves a lot of decisions to the reader. The format that’s easy to implement is the format that gets implemented differently everywhere.

That’s the same sentence I could have written about Markdown, and about CSV. The pattern across this whole series is that a format’s ambiguities don’t stay theoretical. They become somebody’s incident.

Sources
- RFC 8259 — the current JSON standard, and STD 90
- RFC 7493 — I-JSON, the profile that closes the interoperability holes
- ECMA-404 — the parallel Ecma grammar standard
- Nicolas Seriot, “Parsing JSON is a Minefield” — the systematic survey of parser disagreement
- JSONTestSuite — the executable test corpus behind that research, over 300 cases
- CouchDB’s writeup of CVE-2017-12635 — the duplicate-key privilege escalation, in the vendor’s own words
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 20, 2026 Programming Json File-formats Parsing Interoperability

Enjoyed this?
Markdown Is Not a Format, It's an Argument
I’ve covered PNG and text files, and now it’s time for Markdown, which can be thought of as a philosophy of formatting or a lifestyle of text documents more so than an actually well defined file format. It has structure, and it has specifications, plural, and nothing agrees.

Here is three lines of Markdown run through five parsers:
```
INPUT: "- outer\n  - inner\n"

Python-Markdown      <ul> <li>outer</li> <li>inner</li> </ul>
markdown2            <ul> <li>outer <ul> <li>inner</li> </ul></li> </ul>
mistune              <ul> <li>outer<ul> <li>inner</li> </ul> </li> </ul>
marko (CommonMark)   <ul> <li> outer<ul> <li>inner</li> </ul> </li> </ul>
cmark-gfm (GitHub)   <ul> <li>outer <ul> <li>inner</li> </ul> </li> </ul>
```
Five parsers, five different results. Most of that is cosmetic whitespace, but look at the first one: Python-Markdown produced a flat list. The nesting is gone. That’s not a formatting difference, that’s a different document.

The Original Spec Was an Essay

John Gruber released Markdown in March 2004, along with a Perl script called Markdown.pl. The design goal was stated plainly:

The overriding design goal for Markdown’s formatting syntax is to make it as readable as possible. The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions.

That goal was met, and it’s why we’re all still using it twenty years later. The syntax borrowed from conventions people had already invented for plain text email and Usenet: = and - underlines from Setext, # headers from atx, > quoting from Usenet, * for emphasis from Textile and reStructuredText. None of it was new. That was the point.

What Markdown shipped without was a grammar. The specification was English prose describing the syntax with examples, and the tiebreaker for anything the prose didn’t cover was “whatever Markdown.pl does.” A Perl script full of regular expressions became the definition of the format by default.

That works fine until someone writes a second implementation.

Where the Prose Ran Out

The ambiguities weren’t exotic. They were things you hit in the first week:

How much indentation nests a list? Two spaces? Four? One tab? The original prose didn’t say clearly, and the answer interacts with the rule that four spaces means a code block.

What happens inside raw HTML? If you write a <div> and put Markdown inside it, does the Markdown get processed? Gruber’s implementation had behavior; the prose didn’t specify it.

When does a * open emphasis versus just being an asterisk? In a * b * c, are those multiplication signs or emphasis delimiters?

Do underscores work inside words? This one bites daily:
```
INPUT: "snake_case_variable"

Python-Markdown      <p>snake_case_variable</p>
markdown2            <p>snake<em>case</em>variable</p>
mistune              <p>snake_case_variable</p>
marko (CommonMark)   <p>snake_case_variable</p>
cmark-gfm (GitHub)   <p>snake_case_variable</p>
```
markdown2 italicizes your variable name. Every other parser leaves it alone. Both are defensible readings of a spec that never addressed it.

Or the heading with no space after the hash:
```
INPUT: "#Heading"

Python-Markdown      <h1>Heading</h1>
markdown2            <h1>Heading</h1>
mistune              <p>#Heading</p>
marko (CommonMark)   <p>#Heading</p>
cmark-gfm (GitHub)   <p>#Heading</p>
```
Half of them give you a heading, half give you a paragraph starting with a hash. This one matters because #hashtag at the start of a line is a real thing people write.

Everyone Wrote Their Own

With no formal spec, every implementation became a dialect, and the popular ones added features:
- PHP Markdown Extra (Michel Fortin, 2005) added pipe tables, definition lists, footnotes, fenced code blocks, and attribute blocks.
- MultiMarkdown (Fletcher Penney, 2005) added metadata frontmatter, cross-references, citations, and LaTeX export.
- Pandoc Markdown (John MacFarlane, 2006) built a real AST-based parser and added YAML frontmatter, TeX math, grid tables, and citations.
- kramdown (Thomas Leitner, 2009) added inline attribute lists and its own math support.
Each is a superset of a slightly different reading of the original. A document written for one is not guaranteed to render correctly in another, and the failure mode is silent: you don’t get a parse error, you get the wrong document.

CommonMark: Specify the Ambiguity Away

On 3 September 2014, Jeff Atwood announced a spec effort on Coding Horror under the name Standard Markdown, with John MacFarlane as primary author and people from GitHub, Reddit, Stack Exchange, and Meteor involved. The goal was not a new dialect and not a replacement for Gruber’s syntax, but an unambiguous description of what the existing syntax should mean in every case.

The name lasted about a day. That night, by Atwood’s account, Gruber emailed him and MacFarlane privately, called the name “infuriating,” and asked that the project be renamed and the domain taken down. On 4 September, Atwood published a follow-up retitling it Common Markdown, which shortly became the one-word CommonMark.

Worth being precise here, because this story gets retold badly: this was not a trademark action. Gruber holds no registered trademark on “Markdown” and did not invoke one. It was an objection to the name, made in private email, and the only public record of his side is Atwood’s paraphrase. There is no Daring Fireball post about it.

The naming fight is a footnote. The approach is the interesting part. Rather than describing the syntax in prose and hoping, CommonMark defines a parsing algorithm and ships an executable test suite pairing exact input with exact expected HTML, more than 500 examples embedded in the spec document itself. Conformance is not a matter of opinion. You run the tests.

The algorithm works in two passes.

Phase one walks the document line by line and builds block structure. Container blocks (blockquotes, lists, list items) and leaf blocks (headings, code blocks, paragraphs, HTML blocks) get assembled into a tree. Link reference definitions get collected. No inline formatting is considered at all in this phase, which is why block structure always wins: a > at the start of a line is a blockquote marker regardless of what emphasis you thought you were in the middle of.

Phase two walks the text inside leaf blocks and resolves inline structure. This is where emphasis, links, images, code spans, and inline HTML get parsed, using a delimiter stack.

That two-phase split is the single most useful thing to know about Markdown parsing, because it explains most surprising behavior. If your emphasis “leaked” across a list item boundary, it didn’t; blocks were decided before emphasis was ever considered.

The Emphasis Rules Are Hard

Emphasis is the hardest part of the spec, and CommonMark’s solution is a set of flanking rules. A run of * or _ is classified as left-flanking (can open emphasis) or right-flanking (can close it) based on the characters on either side, roughly: a delimiter can open if it’s not followed by whitespace, and can close if it’s not preceded by whitespace, with extra conditions around punctuation.

Then there’s a special case for underscores: an _ can open emphasis only if it’s left-flanking and not right-flanking. That single asymmetry is what makes snake_case_variable safe, because the middle underscores are both left- and right-flanking and are therefore disqualified from opening anything. Asterisks don’t get that rule, which is why snake*case*variable still italicizes.

This is what “specifying the ambiguity away” costs. The rule isn’t elegant. It exists because real documents contain identifiers, and a spec that italicizes your variable names is wrong no matter how clean its grammar is.

You can see the payoff in the nesting case:
```
INPUT: "*foo**bar**baz*"

Python-Markdown      <p><em>foo</em><em>bar</em><em>baz</em></p>
everyone else        <p><em>foo<strong>bar</strong>baz</em></p>
```
Four parsers agree, and the one that predates the delimiter-stack approach gets it wrong in a way that changes the meaning.

GFM Is a Layer, Not a Fork

GitHub Flavored Markdown is CommonMark plus five extensions, and it’s specified against CommonMark rather than diverging from it:
1. Tables, pipe-delimited with alignment colons
2. Task lists, - [ ] and - [x], rendered as checkboxes
3. Strikethrough, ~~text~~
4. Autolinks, bare URLs linkified without brackets
5. A raw HTML filter that neutralizes dangerous tags by escaping their opening bracket
That last one is a security control rather than a formatting feature, which tells you something about what it’s like to run a Markdown renderer on user-submitted content at GitHub’s scale.

The extension boundary is visible if you feed the same table to both:
```
INPUT:
| a | b |
|---|---|
| 1 | 2 |

CommonMark  <p>| a | b | |---|---| | 1 | 2 |</p>
cmark-gfm   <table><thead><tr><th>a</th><th>b</th></tr></thead>...
```
Tables are not Markdown. Tables are a GFM extension. CommonMark renders that input as a paragraph containing literal pipe characters, and it is correct to do so.

Tables, footnotes, task lists, strikethrough, frontmatter, math, and Mermaid diagrams are all extensions. None of them are guaranteed anywhere.

What To Do About It

The practical takeaways are short.

Know which parser you’re targeting. “It renders on GitHub” tells you about cmark-gfm, and nothing about your static site generator, your docs pipeline, or someone’s RSS reader.

Prefer the constructs everyone agrees on. Headings with a space after the hash, fenced code blocks, asterisks for emphasis, blank lines between blocks, four-space or consistent nesting. Boring Markdown survives transport.

Don’t rely on parser-specific behavior you discovered by accident. If nesting a list at two spaces works in your tool, that’s your tool, not the format.

There is even a formal way to say which dialect you mean. RFC 7763 registers text/markdown as a media type, and RFC 7764 defines a variant parameter for exactly this problem:
```
text/markdown; variant=CommonMark
text/markdown; variant=GFM
text/markdown; variant=Original
```
The standards process looked at Markdown, concluded that saying “this is Markdown” is not specific enough to be useful, and standardized a way to say which Markdown you meant.

That’s the tradeoff Markdown made. PNG picked one answer and enforced it with a checksum. A text file refuses to answer anything. Markdown let a million answers bloom, got adopted everywhere precisely because it was easy to implement badly, and has spent the last decade trying to agree with itself.

I’ll take that trade. But it’s worth knowing that when you write Markdown, you are not writing in a format. You’re writing in a dialect, and hoping the reader speaks it.

Sources
- Daring Fireball: Markdown — Gruber’s original 2004 syntax document and design goals
- CommonMark Specification — the parsing algorithm, emphasis flanking rules, and executable test suite
- CommonMark parsing strategy appendix — the two-phase block/inline design
- GitHub Flavored Markdown Spec — the five extensions, specified against CommonMark
- RFC 7763 and RFC 7764 — the text/markdown media type and the registered dialect variants, both by S. Leonard, March 2016
- Coding Horror: Standard Flavored Markdown and Standard Markdown is now Common Markdown — Atwood’s announcement and the rename a day later
- Daring Fireball: Introducing Markdown — the original 15 March 2004 announcement
- tagfilter.c in cmark-gfm — the nine tags GFM’s raw HTML filter neutralizes
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Aug 16, 2026 Programming Markdown File-formats Commonmark Parsing

Enjoyed this?

Parsing

Every XML Feature Is One You Turn Off

Two Kinds of Correct

The Entity System

The Bomb That Got Fixed

What the Rest of It Bought

What To Do About It

Sources

Norway Is Not a Boolean

The Norway Problem

It Was Fixed in 2009

Everything Else That Isn’t a String

Two Ways to Weaponize the Convenience

What To Do About It

Sources

Your JSON Parser Disagrees With Mine

The Same Number, Three Answers

Duplicate Keys Are Legal

NaN Is Not JSON, and Python Emits It Anyway

Why a Small Spec Doesn’t Save You

What To Do About It

Sources

Markdown Is Not a Format, It's an Argument

The Original Spec Was an Essay

Where the Prose Ran Out

Everyone Wrote Their Own

CommonMark: Specify the Ambiguity Away

The Emphasis Rules Are Hard

GFM Is a Layer, Not a Fork

What To Do About It

Sources