Last post ended on a promise: a .docx is a ZIP file, and you already know how ZIP works.

That’s true, and it’s the smaller half of the story. The interesting part is what .docx replaced, because the old .doc format was doing something strange. It wasn’t a document. It was a filesystem with a document living inside it.

A Filesystem in a File

Here’s a real .doc from 2015, 10,240 bytes. The first eight bytes:

d0 cf 11 e0 a1 b1 1a e1

That’s the Compound File Binary Format signature, also called OLE2. file(1) recognizes it and doesn’t even mention Word:

$ file "rich text.doc"
Composite Document File V2 Document, Little Endian, Os: Windows,
Version 1.0, Code page: -535, Revision Number: 0,
Create Time/Date: Thu Dec 10 13:38:22 2015

“Composite Document File” is the honest description. CFBF is a container that implements directories and files, called storages and streams, inside a single flat file. It has a File Allocation Table. It has sectors. If that sounds like FAT16, that’s because it’s the same idea, scaled down to live inside one file on a real filesystem.

Cracking this one open gives:

     106 bytes  CompObj
      20 bytes  Ole
     116 bytes  DocumentSummaryInformation
     312 bytes  SummaryInformation
    2411 bytes  1Table
    3620 bytes  WordDocument

sector size      : 2^9 = 512 bytes
mini sector size : 2^6 = 64 bytes

Two sector sizes, because a 512-byte sector is wasteful for a 20-byte stream. Streams under 4,096 bytes get allocated out of a separate mini-FAT in 64-byte units. There is a fragmentation strategy inside your Word document.

The WordDocument stream is the main event, and it opens with a File Information Block whose magic number is 0xA5EC:

WordDocument stream: 3620 bytes
  FIB magic (wIdent) = 0xA5EC

None of this is the text yet. This is all container.

The Text Is Not in Order

You’d expect the document’s text to sit in the WordDocument stream in reading order. It doesn’t. It sits there in edit order, and a separate structure called a piece table says how to reassemble it.

The piece table is a list of descriptors, each saying “characters at logical position X through Y live at physical offset Z.” Reading a .doc means walking that table and gathering fragments scattered through the stream.

Why build it that way? Because of a feature called Fast Save, and because in 1990 writing to disk was slow. When you edited a document, Word didn’t rewrite the file. It appended your new text to the end of the stream and updated the piece table to point at it. Saving a one-word change to a 200-page document meant writing a few dozen bytes instead of a few hundred kilobytes.

That’s a good optimization. It has an obvious and terrible consequence.

The old text is still in the file. Deleting a paragraph removed it from the piece table, not from the stream. The bytes stayed exactly where they were, unreferenced, invisible in Word, and completely readable in a hex editor.

Microsoft documented this themselves, in a knowledge base article about minimizing metadata in Word documents: “Because of the design of the FastSave feature, text that you delete from a document may remain in the document, even after you save the document.” The recommended fix was to go into Options and clear the “Allow fast saves” check box. From Word 97 SR-1 onward they turned it off by default.

For years, “open the document in a text editor and scroll” was a functioning technique for reading text someone believed they had deleted. Every organization circulating Word files was potentially shipping its own edit history.

The piece table itself has a respectable pedigree. Charles Simonyi brought the technique to Microsoft from Xerox PARC’s Bravo editor, and it’s an elegant way to represent an editable buffer. It’s still how many text editors model documents in memory. The mistake wasn’t the data structure. The mistake was persisting the whole scratch buffer to disk and shipping it to other people.

Then It Became a ZIP of XML

Office 2007 replaced all of it with the Open Packaging Conventions: ECMA-376, later ISO/IEC 29500. A .docx is a ZIP archive containing XML.

Every .docx opens with the same four bytes:

50 4b 03 04    <- PK\x03\x04, a ZIP local file header

PK. Phil Katz’s initials, from the last post, sitting at byte zero of every Word document written since 2007.

Unzip one and the structure is legible:

[Content_Types].xml
_rels/.rels
word/document.xml
word/_rels/document.xml.rels
word/styles.xml
word/settings.xml
word/fontTable.xml
word/theme/theme1.xml
docProps/core.xml
docProps/app.xml

word/document.xml holds the text. [Content_Types].xml maps each part to a MIME type. _rels/.rels is a relationship graph saying which part is the main document and how the parts connect. The whole thing is a tiny website, zipped.

The text itself is WordprocessingML:

<w:p>
  <w:r>
    <w:t>Hello, World!</w:t>
  </w:r>
</w:p>

A paragraph containing a run containing text. Verbose, but you can read it, and more importantly a program you wrote in an afternoon can read it. That is important when building foundational file formats that outlive the creators.

Extracting text from a .doc meant implementing a filesystem and a piece table. Extracting text from a .docx means unzipping and finding <w:t> elements.

The XML contains the document, not the document’s history. Deleted text is deleted.

XML Did Not Mean Simple

It would be tidy to end on “and then it got clean.” The specification runs to several thousand pages, and the ISO fast-track that pushed it through in 2008 was contentious enough to deserve its own post.

What matters here is the shape it settled into. The standard shipped split in two: Strict, the clean format, and Transitional, which carries the legacy baggage forward so documents converted from the binary era still render correctly.

Guess which one nearly everything emits.

Open a Transitional document’s settings and you find a <w:compat> block. Its children are a museum:

w:truncateFontHeightsLikeWP6    WordPerfect 6
w:suppressTopSpacingWP          WordPerfect
w:lineWrapLikeWord6             Word 6
w:autoSpaceLikeWord95           Word 95
w:footnoteLayoutLikeWW8         Word 97
w:useWord97LineBreakRules       Word 97
w:mwSmallCaps                   Mac Word

Every one of those is a flag asking the renderer to reproduce how a specific piece of 1990s software behaved. Not what the format should do. What Word 6 did do, quirks included. Implementing this correctly means emulating applications whose behavior was never written down anywhere.

The bugs were load-bearing, so they got standardized. The format stopped being a filesystem, but it did not stop being a thirty-year-old application’s memory dumped to disk. It just picked a more legible way to write it down.

Which is, in fairness, an enormous improvement. You can read the file now. You just can’t read all of it quickly.

Sources

• MS-CFB: Compound File Binary Format — Microsoft’s spec for the OLE2 container
• MS-DOC: Word Binary File Format — the FIB, the piece table, and the stream layout
• ECMA-376 — Office Open XML, the basis for .docx, and free to download. This is the same specification ISO published as ISO/IEC 29500, so read it here rather than paying ISO for the identical text
• Library of Congress format description for OOXML — preservation notes and format history
• KB Q223790: WD97: How to Minimize Metadata in Word Documents — the fast-save warning, archived; Microsoft no longer hosts it
• w:compat schema reference — the full list of compatibility settings, browsable without downloading the spec

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].