LLBBL Blog

LLBBL Blog

Markdown Is Not a Format, It's an Argument

I’ve covered PNG and text files, and now it’s time for Markdown, which can be thought of as a philosophy of formatting or a lifestyle of text documents more so than an actually well defined file format. It has structure, and it has specifications, plural, and nothing agrees.

Here is three lines of Markdown run through five parsers:

INPUT: "- outer\n  - inner\n"

Python-Markdown      <ul> <li>outer</li> <li>inner</li> </ul>
markdown2            <ul> <li>outer <ul> <li>inner</li> </ul></li> </ul>
mistune              <ul> <li>outer<ul> <li>inner</li> </ul> </li> </ul>
marko (CommonMark)   <ul> <li> outer<ul> <li>inner</li> </ul> </li> </ul>
cmark-gfm (GitHub)   <ul> <li>outer <ul> <li>inner</li> </ul> </li> </ul>

Five parsers, five different results. Most of that is cosmetic whitespace, but look at the first one: Python-Markdown produced a flat list. The nesting is gone. That’s not a formatting difference, that’s a different document.

The Original Spec Was an Essay

John Gruber released Markdown in March 2004, along with a Perl script called Markdown.pl. The design goal was stated plainly:

The overriding design goal for Markdown’s formatting syntax is to make it as readable as possible. The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions.

That goal was met, and it’s why we’re all still using it twenty years later. The syntax borrowed from conventions people had already invented for plain text email and Usenet: = and - underlines from Setext, # headers from atx, > quoting from Usenet, * for emphasis from Textile and reStructuredText. None of it was new. That was the point.

What Markdown shipped without was a grammar. The specification was English prose describing the syntax with examples, and the tiebreaker for anything the prose didn’t cover was “whatever Markdown.pl does.” A Perl script full of regular expressions became the definition of the format by default.

That works fine until someone writes a second implementation.

Where the Prose Ran Out

The ambiguities weren’t exotic. They were things you hit in the first week:

How much indentation nests a list? Two spaces? Four? One tab? The original prose didn’t say clearly, and the answer interacts with the rule that four spaces means a code block.

What happens inside raw HTML? If you write a <div> and put Markdown inside it, does the Markdown get processed? Gruber’s implementation had behavior; the prose didn’t specify it.

When does a * open emphasis versus just being an asterisk? In a * b * c, are those multiplication signs or emphasis delimiters?

Do underscores work inside words? This one bites daily:

INPUT: "snake_case_variable"

Python-Markdown      <p>snake_case_variable</p>
markdown2            <p>snake<em>case</em>variable</p>
mistune              <p>snake_case_variable</p>
marko (CommonMark)   <p>snake_case_variable</p>
cmark-gfm (GitHub)   <p>snake_case_variable</p>

markdown2 italicizes your variable name. Every other parser leaves it alone. Both are defensible readings of a spec that never addressed it.

Or the heading with no space after the hash:

INPUT: "#Heading"

Python-Markdown      <h1>Heading</h1>
markdown2            <h1>Heading</h1>
mistune              <p>#Heading</p>
marko (CommonMark)   <p>#Heading</p>
cmark-gfm (GitHub)   <p>#Heading</p>

Half of them give you a heading, half give you a paragraph starting with a hash. This one matters because #hashtag at the start of a line is a real thing people write.

Everyone Wrote Their Own

With no formal spec, every implementation became a dialect, and the popular ones added features:

  • PHP Markdown Extra (Michel Fortin, 2005) added pipe tables, definition lists, footnotes, fenced code blocks, and attribute blocks.
  • MultiMarkdown (Fletcher Penney, 2005) added metadata frontmatter, cross-references, citations, and LaTeX export.
  • Pandoc Markdown (John MacFarlane, 2006) built a real AST-based parser and added YAML frontmatter, TeX math, grid tables, and citations.
  • kramdown (Thomas Leitner, 2009) added inline attribute lists and its own math support.

Each is a superset of a slightly different reading of the original. A document written for one is not guaranteed to render correctly in another, and the failure mode is silent: you don’t get a parse error, you get the wrong document.

CommonMark: Specify the Ambiguity Away

On 3 September 2014, Jeff Atwood announced a spec effort on Coding Horror under the name Standard Markdown, with John MacFarlane as primary author and people from GitHub, Reddit, Stack Exchange, and Meteor involved. The goal was not a new dialect and not a replacement for Gruber’s syntax, but an unambiguous description of what the existing syntax should mean in every case.

The name lasted about a day. That night, by Atwood’s account, Gruber emailed him and MacFarlane privately, called the name “infuriating,” and asked that the project be renamed and the domain taken down. On 4 September, Atwood published a follow-up retitling it Common Markdown, which shortly became the one-word CommonMark.

Worth being precise here, because this story gets retold badly: this was not a trademark action. Gruber holds no registered trademark on “Markdown” and did not invoke one. It was an objection to the name, made in private email, and the only public record of his side is Atwood’s paraphrase. There is no Daring Fireball post about it.

The naming fight is a footnote. The approach is the interesting part. Rather than describing the syntax in prose and hoping, CommonMark defines a parsing algorithm and ships an executable test suite pairing exact input with exact expected HTML, more than 500 examples embedded in the spec document itself. Conformance is not a matter of opinion. You run the tests.

The algorithm works in two passes.

Phase one walks the document line by line and builds block structure. Container blocks (blockquotes, lists, list items) and leaf blocks (headings, code blocks, paragraphs, HTML blocks) get assembled into a tree. Link reference definitions get collected. No inline formatting is considered at all in this phase, which is why block structure always wins: a > at the start of a line is a blockquote marker regardless of what emphasis you thought you were in the middle of.

Phase two walks the text inside leaf blocks and resolves inline structure. This is where emphasis, links, images, code spans, and inline HTML get parsed, using a delimiter stack.

That two-phase split is the single most useful thing to know about Markdown parsing, because it explains most surprising behavior. If your emphasis “leaked” across a list item boundary, it didn’t; blocks were decided before emphasis was ever considered.

The Emphasis Rules Are Hard

Emphasis is the hardest part of the spec, and CommonMark’s solution is a set of flanking rules. A run of * or _ is classified as left-flanking (can open emphasis) or right-flanking (can close it) based on the characters on either side, roughly: a delimiter can open if it’s not followed by whitespace, and can close if it’s not preceded by whitespace, with extra conditions around punctuation.

Then there’s a special case for underscores: an _ can open emphasis only if it’s left-flanking and not right-flanking. That single asymmetry is what makes snake_case_variable safe, because the middle underscores are both left- and right-flanking and are therefore disqualified from opening anything. Asterisks don’t get that rule, which is why snake*case*variable still italicizes.

This is what “specifying the ambiguity away” costs. The rule isn’t elegant. It exists because real documents contain identifiers, and a spec that italicizes your variable names is wrong no matter how clean its grammar is.

You can see the payoff in the nesting case:

INPUT: "*foo**bar**baz*"

Python-Markdown      <p><em>foo</em><em>bar</em><em>baz</em></p>
everyone else        <p><em>foo<strong>bar</strong>baz</em></p>

Four parsers agree, and the one that predates the delimiter-stack approach gets it wrong in a way that changes the meaning.

GFM Is a Layer, Not a Fork

GitHub Flavored Markdown is CommonMark plus five extensions, and it’s specified against CommonMark rather than diverging from it:

  1. Tables, pipe-delimited with alignment colons
  2. Task lists, - [ ] and - [x], rendered as checkboxes
  3. Strikethrough, ~~text~~
  4. Autolinks, bare URLs linkified without brackets
  5. A raw HTML filter that neutralizes dangerous tags by escaping their opening bracket

That last one is a security control rather than a formatting feature, which tells you something about what it’s like to run a Markdown renderer on user-submitted content at GitHub’s scale.

The extension boundary is visible if you feed the same table to both:

INPUT:
| a | b |
|---|---|
| 1 | 2 |

CommonMark  <p>| a | b | |---|---| | 1 | 2 |</p>
cmark-gfm   <table><thead><tr><th>a</th><th>b</th></tr></thead>...

Tables are not Markdown. Tables are a GFM extension. CommonMark renders that input as a paragraph containing literal pipe characters, and it is correct to do so.

Tables, footnotes, task lists, strikethrough, frontmatter, math, and Mermaid diagrams are all extensions. None of them are guaranteed anywhere.

What To Do About It

The practical takeaways are short.

Know which parser you’re targeting. “It renders on GitHub” tells you about cmark-gfm, and nothing about your static site generator, your docs pipeline, or someone’s RSS reader.

Prefer the constructs everyone agrees on. Headings with a space after the hash, fenced code blocks, asterisks for emphasis, blank lines between blocks, four-space or consistent nesting. Boring Markdown survives transport.

Don’t rely on parser-specific behavior you discovered by accident. If nesting a list at two spaces works in your tool, that’s your tool, not the format.

There is even a formal way to say which dialect you mean. RFC 7763 registers text/markdown as a media type, and RFC 7764 defines a variant parameter for exactly this problem:

text/markdown; variant=CommonMark
text/markdown; variant=GFM
text/markdown; variant=Original

The standards process looked at Markdown, concluded that saying “this is Markdown” is not specific enough to be useful, and standardized a way to say which Markdown you meant.

That’s the tradeoff Markdown made. PNG picked one answer and enforced it with a checksum. A text file refuses to answer anything. Markdown let a million answers bloom, got adopted everywhere precisely because it was easy to implement badly, and has spent the last decade trying to agree with itself.

I’ll take that trade. But it’s worth knowing that when you write Markdown, you are not writing in a format. You’re writing in a dialect, and hoping the reader speaks it.

Sources

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Programming Markdown File-formats Commonmark Parsing