How NTP Keeps the World Synchronized: The Hidden Protocol Behind Every Network Clock

On June 30, 2012, at 23:59:60 UTC, something unusual happened. A single extra second was added to the world’s clocks to account for the Earth’s gradually slowing rotation. Within minutes, Reddit went offline. LinkedIn stopped responding. Mozilla’s servers ground to a halt. Qantas Airways reported that their check-in systems had failed, stranding passengers across Australia.

The culprit wasn’t a cyberattack or a hardware failure. It was a bug in how Linux handled leap seconds—a feature that had been tested only a handful of times in the previous decade. The Network Time Protocol (NTP) had warned servers about the incoming leap second, but the kernel’s high-resolution timer subsystem got confused. Applications that were “sleeping” suddenly woke up all at once, overwhelming CPUs.

This incident revealed something most people never think about: the delicate infrastructure that keeps every computer, phone, and server on the planet synchronized to within milliseconds. That infrastructure is NTP, and it’s been running continuously since before 1985—making it one of the oldest Internet protocols still in use today.

The Accidental Timekeeper

David L. Mills didn’t set out to become the “Father Time” of the Internet. In 1977, he was working at COMSAT Laboratories, trying to synchronize clocks across ARPANET—the precursor to the modern Internet. The problem was fundamental: every computer had its own internal clock, and these clocks drifted apart at rates of seconds per day. When you’re trying to coordinate distributed systems, even small time differences cause chaos.

Mills’ solution evolved over nearly a decade. The first formal specification appeared in RFC 958 in 1985, but the ideas had been circulating since 1981 in Internet Engineering Notes. Version 1 (RFC 1059) arrived in 1988, Version 2 (RFC 1119) in 1989, and Version 3 (RFC 1305) in 1992—a 113-page document that included formal error analysis and a complete error budget.

The current version, NTPv4 (RFC 5905), was published in 2010. It added IPv6 support, improved algorithms, and introduced the Autokey protocol for authentication. Throughout this evolution, Mills maintained the reference implementation and coordinated a global network of volunteer timekeepers until his passing on January 17, 2024, at age 85.

Image source: NTP.org

The Stratum Hierarchy: Time’s Family Tree

NTP doesn’t synchronize everything to a single master clock. Instead, it uses a hierarchical system called “strata” that distributes time from the most accurate sources down to every connected device.

Stratum 0 consists of reference clocks—atomic clocks, GPS receivers, radio time signals. These devices don’t speak NTP directly; they provide reference signals to stratum 1 servers. Think of them as the ultimate authorities on what time it actually is.

Stratum 1 servers are primary time servers. They’re directly connected to stratum 0 devices via serial ports or specialized hardware. These are the most accurate NTP servers available, typically maintained by national standards organizations like NIST (National Institute of Standards and Technology) in the US or PTB (Physikalisch-Technische Bundesanstalt) in Germany.

Stratum 2 through 15 represent increasing distance from the reference clocks. Each “hop” away from stratum 0 introduces additional uncertainty. A stratum 2 server synchronizes to stratum 1 servers; stratum 3 servers synchronize to stratum 2, and so on. The protocol caps at stratum 16, which indicates an unsynchronized device.

Image source: Bodet Time

This hierarchical design serves multiple purposes. First, it prevents a single server from being overwhelmed by billions of clients. Second, it reduces network traffic by allowing local synchronization. Third, it provides redundancy—if one time source fails, clients can fall back to others.

The Four-Timestamp Dance

How does NTP actually synchronize clocks? The elegant answer lies in a simple packet exchange that captures four precise timestamps.

When an NTP client requests the time from a server, it records the exact moment it sent the request (T1). The server receives this request and records that moment (T2), then sends a response, recording the transmission time (T3). Finally, the client receives the response and records that moment (T4).

From these four timestamps, NTP calculates two critical values:

Clock Offset (how much the client’s clock differs from the server’s):

Round-trip Delay (how long the network transmission took):

The offset calculation assumes symmetric network delays—the trip from client to server takes the same time as the return trip. This assumption rarely holds perfectly in practice, which is why NTP uses sophisticated filtering algorithms. The protocol selects the offset from the packet with the smallest delay out of the last eight exchanges, assuming that shorter delays indicate less asymmetric conditions.

sequenceDiagram
    participant Client
    participant Server
    Note over Client: T1: Send request
    Client->>Server: NTP Request (T1)
    Note over Server: T2: Receive request
    Note over Server: T3: Send response
    Server->>Client: NTP Response (T1, T2, T3)
    Note over Client: T4: Receive response
    Note over Client: Calculate offset and delay

The 48 Bytes That Keep the World on Time

Every NTP packet is exactly 48 bytes—compact enough to fit in a single UDP packet, simple enough to process quickly, yet sophisticated enough to carry all necessary information.

The first byte contains three critical fields: LI (Leap Indicator, 2 bits), VN (Version Number, 3 bits), and Mode (3 bits). The Mode field determines whether the packet is from a client (3), server (4), or operating in symmetric peer mode (1 or 2).

Next comes the Stratum field (1 byte), which tells the receiver how many hops away this server is from a reference clock. The Poll field (1 byte) indicates the interval between successive messages, measured as a power of 2 in seconds. The Precision field (1 byte) describes the precision of the local clock, also as a power of 2.

Four critical 32-bit fields follow: Root Delay (total round-trip delay to the primary reference source), Root Dispersion (the maximum error relative to the primary reference), and Reference Identifier (identifying the reference source).

Then come the four 64-bit timestamps: Reference Timestamp (when the clock was last set), Originate Timestamp (T1, when the client sent its request), Receive Timestamp (T2, when the server received the request), and Transmit Timestamp (T3, when the server sent its response).

This structure has remained remarkably stable since the early versions of NTP, a testament to Mills’ thoughtful design.

Clock Discipline: The Hybrid PLL/FLL Algorithm

Knowing the time offset is only half the battle. The real challenge is adjusting the local clock to match the reference time without causing disruption. This is where NTP’s clock discipline algorithm comes in—a hybrid phase-locked loop (PLL) and frequency-locked loop (FLL) system.

Computer clocks are notoriously inaccurate. A typical quartz crystal oscillator might drift by several hundred parts per million (PPM)—several seconds per day. Temperature changes, component aging, and manufacturing tolerances all contribute to this drift. Simply stepping the clock forward or backward to match the reference time would cause problems: applications might see time go backward, or intervals might be miscalculated.

Instead, NTP gradually adjusts both the time and the frequency of the local clock. The PLL mode handles short-term phase corrections, ideal when poll intervals are short (below the Allan intercept, typically around 2048 seconds). The FLL mode handles long-term frequency corrections, better suited for longer poll intervals where frequency stability matters more than phase alignment.

The algorithm also includes a “popcorn spike suppressor”—nonlinear signal processing code that filters out occasional large jitter spikes that would otherwise throw off the synchronization. This was developed after early experiences with transatlantic satellite links where jitter could exceed one second.

The result? With modern hardware and a precision pulse-per-second (PPS) signal, NTP can achieve synchronization accuracy in the tens of nanoseconds. For typical Internet synchronization, accuracy ranges from a few milliseconds on local networks to tens of milliseconds across the global Internet.

The Leap Second Problem

Leap seconds exist because the Earth’s rotation isn’t perfectly constant. Tidal forces, atmospheric circulation, and molten metal flow in the Earth’s core all cause the planet’s rotation to speed up or slow down unpredictably. The International Earth Rotation and Reference Systems Service (IERS) announces leap seconds about six months in advance when the discrepancy between UTC (Coordinated Universal Time) and UT1 (astronomical time) approaches 0.9 seconds.

Since 1972, 27 leap seconds have been added. The last one occurred on December 31, 2016. But between 1999 and 2005, there were no leap seconds at all—a seven-year gap during which much of modern cloud infrastructure was developed and deployed.

This gap proved costly. When the 2012 leap second arrived, systems that had never experienced one suddenly had to handle a time value they weren’t designed for: 23:59:60.

The Linux kernel’s hrtimer (high-resolution timer) subsystem wasn’t prepared. When the leap second was inserted, timers that were set to expire at a certain time found themselves a second behind. The kernel responded by waking up all sleeping applications simultaneously, overwhelming CPUs.

Reddit’s Cassandra database servers locked up. LinkedIn’s Tomcat servers failed. Mozilla’s Hadoop clusters crashed. Qantas Airways’ booking systems went down. The recovery took hours, and the postmortems revealed a fundamental problem: code that handles leap seconds is rarely executed, rarely tested, and full of edge cases.

Google’s solution, developed after experiencing similar problems in 2008 and 2012, became known as “leap smear.” Instead of inserting a single second at once, Google’s NTP servers gradually slow down their clocks by about 0.0014% over a 20-hour window centered on the leap second. The effect is that clients never see a repeated second or a jump backward—they simply experience slightly slower time for a day.

Image source: Google Cloud Blog

The Security Problem: NTP as an Attack Vector

For most of its history, NTP was designed for a trusted Internet. Authentication existed but wasn’t widely deployed. This naivety came back to bite the Internet in 2013 and 2014, when NTP amplification attacks became a major threat.

The vulnerability was in the monlist command, a diagnostic feature that returns information about the last 600 hosts that contacted the server. A 234-byte request could trigger a response of up to 4,680 bytes—a 100x amplification factor. Attackers could spoof the source IP address of their victim, causing vulnerable NTP servers to flood the target with traffic.

In February 2014, a massive NTP amplification attack against Cloudflare peaked at 400 Gbps—the largest DDoS attack ever recorded at the time. The attack exploited default configurations of older NTP servers that had monlist enabled.

The response was twofold. First, server operators disabled monlist or upgraded to versions that restricted it. Second, the IETF developed Network Time Security (NTS), published as RFC 8915 in September 2020.

NTS uses TLS 1.3 to establish an authenticated connection and exchange keys, then uses those keys to authenticate NTP packets with AEAD (Authenticated Encryption with Associated Data). This prevents man-in-the-middle attacks where an attacker could intercept and modify NTP packets to skew the victim’s clock.

The security implications of time manipulation go beyond denial of service. Attackers who can manipulate a system’s clock can:

• Cause SSL/TLS certificates to appear expired or not yet valid
• Bypass time-based access controls
• Corrupt database logs and audit trails
• Trigger race conditions in distributed systems
• Cause certificate revocation checks to fail

This makes NTS deployment critical for security-sensitive environments.

Modern Implementations: chrony vs. ntpd

The classic NTP implementation, ntpd, has been around since the protocol’s inception. But modern Linux distributions increasingly default to chrony, a newer implementation with several advantages:

Faster convergence: chrony can synchronize a clock in seconds, compared to minutes for ntpd. This matters for systems that boot frequently or experience significant clock drift.

Better handling of intermittent connections: chrony was designed for systems that may be offline or have unstable network connections—laptops, mobile devices, and VMs.

Hardware timestamping: On Linux, chrony supports hardware timestamping for extremely accurate local network synchronization.

More responsive to large steps: When the clock is significantly off, chrony can make larger adjustments more safely.

The ntpsec project is another modern fork that focuses on security hardening, removing legacy code and potential vulnerabilities from the original ntpd codebase.

For most deployments, chrony is now the recommended choice. It’s the default in RHEL 7+, CentOS 7+, Fedora, and Ubuntu 17.04+.

NTP vs. PTP: When Milliseconds Aren’t Enough

NTP typically achieves synchronization accuracy of 1-10 milliseconds on local networks and 10-100 milliseconds across the Internet. For most applications, this is sufficient. But for financial trading, telecommunications, industrial automation, and scientific experiments, microseconds or even nanoseconds matter.

This is where Precision Time Protocol (PTP, IEEE 1588) comes in. PTP can achieve sub-microsecond accuracy by:

• Using hardware timestamping at the network interface level
• Eliminating software processing delays from the time calculation
• Assuming a controlled network environment where delays are predictable

PTP works best in managed networks where switches and routers support PTP transparently. NTP, by contrast, is designed for the chaotic Internet where delays vary unpredictably.

The choice between NTP and PTP depends on your requirements:

The NTP Pool: Volunteer Time for the World

With billions of devices needing time synchronization, no single organization could possibly serve all of them. The solution is the NTP Pool, a volunteer-run distributed system of time servers accessible through DNS names like pool.ntp.org, us.pool.ntp.org, or asia.pool.ntp.org.

When a device queries pool.ntp.org, DNS returns a rotating set of IP addresses from volunteers who have contributed their servers to the pool. This provides automatic load balancing and geographic distribution.

As of 2024, the NTP Pool includes over 4,000 servers worldwide, serving billions of requests per day. Operating a stratum 1 or stratum 2 server in the pool is a way for organizations to contribute to Internet infrastructure.

Practical Recommendations

For most deployments, follow these best practices:

1. Use multiple time sources: At least three servers protect against one being wrong or unavailable. NTP’s intersection algorithm will detect and reject a falseticker.

2. Don’t connect directly to stratum 1 servers: They’re reserved for stratum 2 public servers. Connect to the pool or to stratum 2/3 servers instead.

3. Prefer chrony over ntpd for modern Linux systems.

4. Use NTS when available for security-sensitive deployments.

5. Don’t mix smearing and non-smearing servers: Google’s time servers use leap smear; most others don’t. Mixing them will cause inconsistent behavior during leap seconds.

6. Monitor your synchronization: Tools like ntpq and chronyc can show you the offset, delay, and jitter of your time sources.

The Invisible Infrastructure

NTP runs quietly in the background on virtually every connected device. It’s so ubiquitous that most people never think about it—until something goes wrong. Financial markets depend on it for trade timestamping. Power grids use it to coordinate across thousands of miles. Cellular networks rely on it for handoffs between towers. Database replication requires it for conflict resolution.

The protocol that David Mills designed in the 1980s has scaled from hundreds of servers to billions of devices. Its hierarchical architecture distributes load while maintaining accuracy. Its algorithms handle the chaotic reality of Internet latency. And its recent security enhancements address the challenges of a less trusting world.

When your phone automatically adjusts for daylight saving time, when your laptop’s clock matches your colleague’s across the world, when a financial trade is timestamped for regulatory compliance—NTP is working behind the scenes, keeping the world synchronized one millisecond at a time.

References

1. Mills, D.L. “Network Time Protocol (Version 3) Specification, Implementation and Analysis.” RFC 1305, IETF, March 1992. https://datatracker.ietf.org/doc/html/rfc1305

2. Mills, D.L., et al. “Network Time Protocol Version 4: Protocol and Algorithms Specification.” RFC 5905, IETF, June 2010. https://datatracker.ietf.org/doc/html/rfc5905

3. Mills, D.L. “A Brief History of NTP Time: Confessions of an Internet Timekeeper.” University of Delaware. https://www.ntp.org/reflib/papers/history.pdf

4. Franke, D., et al. “Network Time Security for the Network Time Protocol.” RFC 8915, IETF, September 2020. https://datatracker.ietf.org/doc/html/rfc8915

5. McKeown, A. “The Inside Story of the Extra Second That Crashed the Web.” Wired, July 2, 2012. https://www.wired.com/2012/07/leap-second-glitch-explained/

6. Shields, M. “Making every (leap) second count with our new public NTP servers.” Google Cloud Blog, November 30, 2016. https://cloud.google.com/blog/products/gcp/making-every-leap-second-count-with-our-new-public-ntp-servers

7. “NTP Amplification Attacks Using CVE-2013-5211.” CISA Alert, January 13, 2014. https://www.cisa.gov/news-events/alerts/2014/01/13/ntp-amplification-attacks-using-cve-2013-5211

8. “Comparison of NTP implementations.” chrony Project. https://chrony-project.org/comparison.html

9. “Structure of the NTP Data packet.” Meinberg. https://www.meinbergglobal.com/english/info/ntp-packet.htm

10. “Stratums in the NTP Protocol: Understanding the hierarchy of servers.” Bodet Time, March 4, 2024. https://www.bodet-time.com/resources/blog/1882-stratums-in-the-ntp-protocol-understanding-the-hierarchy-of-servers.html