XML’s answer to structure was a schema language, a query language, and a namespace system. MP4’s answer is eight bytes, and it turned out to be enough to absorb an entire industry. Every byte in the file lives inside a box, and every box starts the same way:

4 bytes   size, big-endian, including this header
4 bytes   type, four ASCII characters

That is it. Eight bytes, and any parser that knows nothing else about the format can walk the entire file, skipping what it doesn’t understand. Some boxes contain other boxes. Some contain payload. There is no data outside a box anywhere in the file.

That design is why the container outgrew video entirely.

---

Same Header, Different Extension

Here is a ten-second 640x360 H.264 clip:

ftyp  32 bytes  @ 0
free  8 bytes  @ 32
mdat  63602 bytes  @ 40
moov  4424 bytes  @ 63642

Here is a three-second AAC audio file:

ftyp  28 bytes  @ 0
free  8 bytes  @ 28
mdat  26304 bytes  @ 36
moov  1287 bytes  @ 26340

And here is a still image:

ftyp  32 bytes  @ 0
meta  235 bytes  @ 32
  hdlr  33 bytes  @ 44
  pitm  14 bytes  @ 77
  iloc  30 bytes  @ 91
  iinf  40 bytes  @ 121
  iprp  106 bytes  @ 161
    ipco  75 bytes  @ 169
    ipma  23 bytes  @ 244
mdat  17667 bytes  @ 267

That last one is an AVIF. A photograph. It has the same box header format, the same ftyp first, the same mdat holding the payload. What changed is that a still image has no timeline, so instead of moov with its sample tables it uses meta with an item structure: pitm names the primary item, iloc says where in mdat that item’s bytes live, iprp carries its properties.

The ftyp box says which dialect you are reading:

$ xxd -g 1 -l 32 shot.avif
00000000: 00 00 00 20 66 74 79 70 61 76 69 66 00 00 00 00  ... ftypavif....
00000010: 61 76 69 66 6d 69 66 31 6d 69 61 66 4d 41 31 42  avifmif1miafMA1B

Size 32, type ftyp, major brand avif, then four compatible brands: avif, mif1, miaf, MA1B. The .m4a file above declares M4A with isom as a compatible brand. The MP4 declares isom with iso2.

So .mp4, .mov, .m4a, .m4v, .heic, and .avif are one format with six extensions. Your iPhone photo library and your video library are the same container. That happened because in February 1998, ISO picked Apple’s QuickTime file format as the basis for MPEG-4’s container, and the box model turned out to be general enough that everyone who needed a container afterward just used it.

---

The Box in the Wrong Place

Now the flaw that shaped an entire decade of web video.

The moov box holds the sample tables: which frame starts at which byte, how long it lasts, which chunk it belongs to. mdat holds the frames. A player can decode nothing until it has read moov, because mdat has no internal framing at all. It is one undifferentiated run of bytes, and the only thing that says where frame 0 begins is a number inside moov.

Look at where moov ended up in that first file. Byte 63,642 of a 68,066-byte file.

An encoder writing sequentially cannot know the byte offset of the last chunk until it has written the last chunk, so the natural thing is to write all of mdat and then append moov at the end. That is what nearly every encoder did by default, and it means the player must reach the last 6% of the file before it can show you the first frame.

The fix is a post-processing pass:

$ ffmpeg -i input -c:v libx264 -movflags +faststart out.mp4

ftyp  32 bytes  @ 0
moov  4424 bytes  @ 32
free  8 bytes  @ 4456
mdat  63602 bytes  @ 4464

Same boxes. Same sizes. Same total file length, 68,066 bytes both times, and I checked the mdat payloads byte for byte: identical. All that changed is the order.

It is not quite a memmove, though, because moov’s offsets are absolute positions in the file:

plain.mp4    stco has 1 chunk offsets; first five: [48]
fast.mp4     stco has 1 chunk offsets; first five: [4472]

Moving moov in front of mdat pushed every byte of media 4,424 places later, so every entry in the chunk offset table had to be rewritten by exactly that amount. On a real file with thousands of chunks, that is thousands of pointers, all of which must be corrected, and if the rewrite changes the size of moov (32-bit offsets overflowing into co64) the whole thing has to be recomputed again.

Absolute offsets are the design decision underneath most of MP4’s awkwardness. You cannot concatenate two MP4s. You cannot insert a second of video in the middle. You cannot append to a file that is still being written and have it remain playable. Everything is pointer arithmetic against byte zero.

---

Fragments Are the Actual Answer

Fragmented MP4 fixes it by giving up on the single index. Instead of one moov describing the whole timeline, you get an initialization segment and then a run of self-describing chunks:

[ ftyp + moov ]  [ moof + mdat ]  [ moof + mdat ]  [ moof + mdat ] ...

Each moof carries the sample table for the mdat that follows it, with offsets relative to the fragment rather than the file. Which means you can start writing before you know how long the video is, cut the stream anywhere, serve any fragment independently, and switch bitrates between fragments without the player noticing.

That property is the entire basis of HLS and DASH. Every adaptive-bitrate stream you have watched is this: a manifest, an init segment, and a pile of moof/mdat pairs that a player stitches together while quietly swapping quality levels based on your bandwidth.

It also means the moov placement problem is now mostly historical for streaming and still completely current for files. Anything you upload, download, or store as a single .mp4 still has one moov, and it is still in whichever place the encoder happened to put it.

---

What To Do About It

• Always pass -movflags +faststart when producing MP4 for the web. It costs one extra pass over the file at encode time and nothing at all afterward.
• Check where moov landed before blaming the network. Eight bytes of parsing tells you: read the size at offset 0, jump, read the type, repeat. If moov is last, that is your slow start.
• Use fMP4 for anything live or adaptive. A single-file MP4 cannot be written and played at the same time, no matter how you order the boxes.
• Don’t concatenate MP4 files. cat a.mp4 b.mp4 > c.mp4 produces a file whose first moov describes only the first video and whose second moov has offsets pointing into the wrong place. Remux with ffmpeg -f concat instead.
• Treat .heic and .avif as the same problem space. If your image pipeline calls identify or sniffs magic bytes, those files start with a box header, not a signature, and the four bytes that matter are at offset 4 rather than offset 0.
• Read ftyp compatible brands, not the extension. A file named .mp4 can declare qt , and a file named .mov can declare isom. The brands are the truth.

Everything in this series so far has failed by underspecifying something. MP4 does not have that problem. The box model is rigorous, self-describing, and general enough that it absorbed still images without anyone having to redesign it. What it got wrong was one thing: it wrote down byte offsets instead of relative ones, and made the index a single object that has to be complete before it can be written. Two decades of streaming infrastructure exist to work around that decision.

Sources

• ISO/IEC 14496-12 — the ISO Base Media File Format; the box definition is clause 4.2
• MP4 Registration Authority — the registry of every legal FourCC brand and box type
• Apple’s QuickTime File Format documentation — the atom model MP4 inherited
• AVIF specification — how AV1 intra frames map onto ISOBMFF items
• RFC 8216 — HTTP Live Streaming, which is fMP4 plus a text manifest
• ffmpeg movflags documentation — +faststart and the fragmentation options

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].