LLBBL Blog

LLBBL Blog

Day 15: The Man Who Synchronized the World

David Mills, the father of internet time, wrote the protocol that synchronizes every computer on Earth. He did it as a professor at the University of Delaware, on a project he started in the early 1980s and never stopped working on.

The code lives in your laptop, your phone, your router, every cloud server you’ve ever touched, and every satellite in low Earth orbit. The protocol it implements is called NTP. The reason your computer’s clock is correct, right now, within a few milliseconds of UTC, is that Mills spent forty years of his life making sure it would be.

He once described the early ARPANET days as a “sandbox” where researchers were simply told to “do good deeds.” Part of the allure of the time-synchronization work, he told The New Yorker in 2022, was that he was just about the only one doing it. He had his own “little fief.”

For forty years, that is exactly what it was.

The problem

The early internet had a clock problem. As soon as there were enough machines on the network that “what time is it?” didn’t have a single answer, somebody was going to have to write a protocol. Each computer had its own oscillator. Each oscillator drifted at its own rate. Two machines that agreed at noon could be tens of seconds apart by midnight.

Why did this matter? For most things, it didn’t. For some things, it mattered a lot. A file saved on one machine and copied to another could look older than the version it overwrote, confusing every backup tool that assumed time moves forward. Cryptographic handshakes that expire after a few seconds could fail because the two ends disagreed on what “a few seconds ago” meant. Database replicas could apply writes in the wrong order and corrupt their own state. Email between two servers could arrive timestamped before it was sent. Debugging a multi-machine bug meant correlating log entries across clocks that didn’t agree about which event came first.

Mills decided the actual problem was that there was no protocol for negotiating the truth (in this case, time) across multiple systems. The clock on his desk was wrong. Every other clock was also wrong. The question wasn’t “who has the right time?", it was “given that nobody has the right time and the network adds an unknown delay to every measurement, how does the system converge on a consensus that is closer to UTC than any individual node could achieve alone?"

His first NTP RFC, RFC 958, was published in September 1985. We now call that protocol NTPv0, or the prototype. In it, Mills nailed down the four-timestamp packet format and the offset/delay math that has been in every revision since. The packet format and the core algorithm haven’t meaningfully changed in forty years. That kind of staying power is rare in any field. In internet infrastructure, where the half-life of a protocol can be measured in single-digit years, it is quite commendable.

The four timestamps

NTP’s core insight is that the network delay between client and server can be measured, not just guessed, as long as both sides record their own timestamps for both legs of the conversation. Four timestamps are exchanged in a single round trip:

Client                            Server
──────                            ──────
T₁ ──── request ───────────────►
                                  T₂
                                  T₃
T₄ ◄────────────── response ─────
  • T₁ — the client sends the request (client clock)
  • T₂ — the server receives it (server clock)
  • T₃ — the server sends the response (server clock)
  • T₄ — the client receives it (client clock)

Now the client has four numbers. T₁ and T₄ are in the client’s reference frame, T₂ and T₃ are in the server’s. From those four numbers, two things fall out: the round-trip delay (how long the conversation took, minus the time the server spent thinking) and the clock offset (how far the client’s clock is from the server’s). The client now knows how wrong it is, and by how much.

The math depends on one critical assumption: the network is symmetric. The packet takes the same time to travel in both directions.

If you’ve been following along in the series, you know there are a lot of ways to measure time. Atomic clocks. GPS receivers. The quartz crystal in your laptop. Radio signals broadcast from government antennas. They don’t all tick at the same rate, and they don’t all agree on what the current time is. How does NTP reconcile across that much varity in time sources?

The stratum hierarchy

NTP organizes the world’s clocks into a tree, with depth measured in strata.

Stratum 0 is the reference. Cesium atomic clocks. Hydrogen masers. GPS receivers. Radio receivers tuned to WWV, DCF77, or MSF. These are not on the network, they’re physical devices wired directly to a small number of computers via PPS pulses on serial ports.

Stratum 1 is the small group of servers wired directly to Stratum 0. There are perhaps a few thousand of these globally. NIST runs some. Major universities run some. The big internet exchanges run some.

Stratum 2 servers sync with Stratum 1, Stratum 3 with Stratum 2, and so on down to Stratum 15. Stratum 16 means “unsynchronized, do not trust.”

A typical Linux laptop syncs against Stratum 2 or 3 servers. A typical cloud VM syncs against its provider’s internal Stratum 1 fleet. Your phone syncs against whatever its carrier provides. The whole tree is held together by NTP itself, recursively.

The genius of the design is that there is no central authority. Mills did not own the protocol. There is no “official NTP server.” Anyone can run a Stratum 1 with the right hardware, and anyone can run a Stratum 2+ by syncing with a few Stratum 1s of their choice. The largest public pool, pool.ntp.org, is a volunteer effort started in 2003 by Adrian von Bidder. It currently aggregates a few thousand donated stratum-2 servers worldwide and serves several billion requests per day. Nobody is in charge of it. It just works.

The slew, not the step

There are three different times to keep track of on every synced computer. The reference time is what UTC says, the truth NTP is chasing. The tick rate is how fast the computer’s oscillator pulses. It’s supposed to produce one second of clock time per real second, but always drifts a little. The system clock is what gets reported when an application asks for the current time. Synchronizing means closing the gap between the system clock and the reference time without breaking anything that depends on the system clock being well-behaved.

NTP’s primary tool for that is the slew: it adjusts the tick rate, making each tick slightly longer or shorter than nominal, so the system clock drifts into alignment on its own. The alternative would be to jump the clock forward or backward by the full offset (a step), which is fast but can produce duplicate keys in a database, expire valid TLS sessions, or cause a logging system to mis-order events.

Mills designed ntpd to slew conservatively. A 200ms gap might take several minutes to close, and corrections larger than about 128ms would get stepped because slewing them gradually was prohibitively slow. That trade-off worked for the always-on Unix workstations of the 1980s and 90s. It works less well for the modern reality of laptops that suspend for hours and resume with a clock that hasn’t been touched since last Tuesday, or cloud VMs that get migrated between hosts. Modern variants like chrony slew more aggressively for exactly that reason. When you open your laptop lid, you want the clock right now, not after fifteen minutes of imperceptible easing.

The legacy

In a sense, NTP is the thing that made the modern internet possible.

Without well-synchronized clocks, you cannot have SSL certs. The browser needs to know when the cert expires, and if its clock is off by more than a few minutes, the encryption breaks. The same goes for databases. No matter the type, NoSQL or otherwise, they all depend on a clock to record when an operation took place.

Without NTP, cell towers wouldn’t agree on when to hand off a call. Financial transactions wouldn’t be enforceable. And all those log files you’ll totally read one day wouldn’t make any sense. NTP is foundational to all of it. It runs as a daemon on every machine, the ones you stare at all day, the ones you don’t see, and the ones you don’t care about.

We remember Mills as the internet’s “Father Time” and the man who synchronized the world. Neither is a metaphor.

Sources

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Tomorrow: ISO 8601, the format wars, the carnage of MM/DD vs DD/MM, and why 2026-06-07T14:30:00Z won.

/ DevOps / Programming / 30daysoftime / Internet history / Protocols