The Second That Broke the Internet: How One Extra Tick of the Clock Took Down Reddit, Airlines and DNS

The Second That Shouldn't Exist#
It's Sunday morning in Australia. You're at the airport with a bag to check and a flight to catch. The check-in system stops working. The queue behind you grows. Staff switch to doing check-in by hand.
At the same moment, on the other side of the world, it's Saturday evening in California. Engineers at Mozilla are looking at a graph that makes no sense: CPU usage on their servers has shot to the ceiling, and nothing unusual is running. Reddit is down.
In India it's just before 5:30 AM on 1 July 2012. Something strange has just happened to every clock that keeps official time on Earth.
The day of 30 June 2012 had one extra second. Official clocks didn't go from 23:59:59 to 00:00:00. They went 23:59:59, then 23:59:60, then 00:00:00. That one extra second is called a leap second, and on that night a bug in how Linux handled it pinned servers at full CPU across the internet.
Four and a half years later, another leap second hit. On New Year's Day 2017, inside Cloudflare's DNS software, time briefly ran backwards, a number that should never be negative went negative, and some DNS lookups for Cloudflare customers started failing.
This case file explains why one second is so dangerous, and why the world has voted to stop adding them.
A leap second adds 23:59:60 to keep UTC within 0.9 seconds of the Earth's spin, and most computers repeat 23:59:59 instead. In 2012 Linux timers ran a second ahead and fired forever, taking down Reddit for 30 to 40 minutes and delaying 400+ Qantas flights; in 2017 Cloudflare measured a negative rtt and some DNS lookups failed. The fixes are monotonic clocks for durations, smearing the second, and the CGPM's 2022 vote to stop adding leap seconds by 2035.
What Is a Leap Second?#
Humans have two ways of measuring a day, and they don't quite agree.
The first is atomic clocks. They count seconds using the vibrations of atoms, and they are astonishingly steady. A timescale built purely from atomic clocks is called TAI (International Atomic Time). It never skips, never repeats.
The second is the Earth itself. A day is one spin of the planet, and the timescale that follows the spin is called UT1. The problem is that the Earth is a wobbly clock. Its spin speeds up and slows down by tiny amounts, thanks to tides, the planet's core, even ice melting and refreezing.
The time on your phone and on every server is UTC (Coordinated Universal Time). UTC ticks at the atomic rate, but it's kept within 0.9 seconds of the Earth's spin. When the gap between them grows too big, an organisation called the IERS (International Earth Rotation and Reference Systems Service) announces a leap second about six months in advance. So far they have always come at the end of June or December.
%% title: Where UTC comes from
flowchart TD
A["Atomic clocks<br/>perfectly steady seconds"]:::store --> TAI["TAI · atomic time<br/>never skips"]:::service
E["The Earth's spin<br/>a slightly wobbly clock"]:::external --> UT1["UT1 · Earth time"]:::service
TAI --> CMP{"IERS watches the gap<br/>UT1 minus UTC"}:::hot
UT1 --> CMP
CMP -->|"gap approaching 0.9 s"| LS["Announce a leap second<br/>about six months ahead"]:::core
LS --> UTC["UTC · the time everyone uses<br/>atomic ticks plus leap seconds"]:::core
UTC --> NTP["Time servers · NTP"]:::edge
NTP --> S(["Your servers and phone"]):::clientSince the first one on 30 June 1972, 27 leap seconds have been added. All of them were positive: an extra second inserted. The last was on 31 December 2016.
| Term | Plain meaning |
|---|---|
| TAI | Time from atomic clocks only. Steady, no leap seconds |
| UT1 | Time from the Earth's spin. Wobbly |
| UTC | The world's official time: atomic ticks, nudged by leap seconds to stay near UT1 |
| Leap second | An extra second (23:59:60) added to UTC to keep it close to UT1 |
| NTP | Network Time Protocol: how computers ask time servers what time it is |
| Wall clock | The computer's idea of the current date and time. Can be changed and can jump |
| Monotonic clock | A counter that only ever moves forward. Useless for dates, perfect for measuring durations |
| Smear | Spreading a leap second over many hours by running clocks very slightly slow |
Two Clocks Inside Every Computer#
Your computer doesn't have one clock. It has (at least) two, and they answer different questions.
The wall clock answers "what time is it?" It's the date and time you see on screen and in logs. It's set from the network, it can be corrected, and it can jump: forwards or backwards. When your laptop wakes up and syncs with a time server, its wall clock might jump. When a leap second arrives, most computers can't represent 23:59:60, so they repeat 23:59:59. Time on the wall clock goes back by one second.
The monotonic clock answers "how much time has passed?" It starts at some arbitrary number when the machine boots and only ever counts up. It doesn't know the date. It never jumps backwards.
Russ Cox of the Go team put it neatly in Go's own design document: "The wall clock is for telling time; the monotonic clock is for measuring time."
Wall clock
- "What time is it?"
- Shows dates, goes in logs and receipts
- Can be set, corrected and jump backwards or forwards
- Go
time.Now()(wall part) · Pythontime.time()· JavaSystem.currentTimeMillis()· JavaScriptDate.now()
Monotonic clock
- "How long did that take?"
- Measures durations, timeouts, retries, rate limits
- Never goes backwards
- Go 1.9+
time.Since(start)· Pythontime.monotonic()· JavaSystem.nanoTime()· JavaScriptperformance.now()
The bug in both of our stories is the same mistake at heart: using the wall clock to measure time. Look at what happens when you subtract two wall-clock readings across a leap second. The example below comes straight from Go's design document.
- First reading, just before the repeat: 23:59:59.995
- 10 ms later the clock has been set back one second, so the second reading is: 23:59:59.005
- Elapsed = 23:59:59.005 − 23:59:59.995 = −990 ms instead of +10 ms
Negative time. Code that assumes "elapsed time is never negative" (which is almost all code) is now in territory nobody tested.
How the leap second reaches your server: NTP#
Computers learn the time with NTP (Network Time Protocol). Your server asks a time server "what time is it?", and adjusts its wall clock a little at a time. NTP packets also carry a two-bit leap indicator. A value of 1 means "the last minute of today has 61 seconds." That's the warning. At midnight, the operating system does the insertion itself.
%% title: How a leap second is delivered (simplified, illustrative times) sequenceDiagram autonumber participant T as NTP time server participant K as Your server's kernel participant A as Your application T->>K: Time is 23:30:00 · leap indicator = 1 Note over K: "Tonight's last minute has 61 seconds" A->>K: What time is it? (wall clock) K-->>A: 23:59:59.995 Note over K: Leap second · kernel sets the clock back 1 s A->>K: What time is it? (wall clock) K-->>A: 23:59:59.005 · earlier than before A->>K: How long since boot? (monotonic clock) K-->>A: Still counting up · unaffected
On Linux, this is when you'd see a line like Clock: inserting leap second 23:59:60 UTC in your logs. In 2012, that line was the last calm moment for a lot of servers.
2012: The Timer That Fired Forever#
- 23:59:60 UTC — the leap second. That's 05:29:60 IST in India and just before 10 AM on Sunday in eastern Australia
- 00:00 UTC (17:00 Pacific) — Mozilla's MySQL and Java servers jump to very high CPU with nothing unusual running
- Around the same time — Reddit goes offline for 30 to 40 minutes; LinkedIn, Gawker, Foursquare and Yelp also report problems
- Sunday morning in Australia — the Amadeus Altea reservation system, used by Qantas and Virgin Australia, fails for about an hour. More than 400 Qantas flights are delayed as staff fall back to manual check-in
- ~02:35 UTC (19:35 Pacific) — Mozilla fixes its servers by setting the date, and rolls the fix out with Puppet
- Following days — kernel developer John Stultz tracks down the bug and posts a fix
What actually went wrong inside Linux#
The Linux kernel keeps the wall-clock time. To insert a leap second, it sets that clock back one second as the leap second begins, so 23:59:59 is repeated.
But the kernel has another part, the high-resolution timer (hrtimer) system, which wakes programs up at precise moments ("wake me in 200 milliseconds"). It keeps its own small offsets from the system time. Whenever the system time changes, the kernel is supposed to call a function named clock_was_set() so the timers can update those offsets.
After the leap second, that call didn't happen. As LWN's Jonathan Corbet explained, the timer system's idea of the current time had moved forward while the system time was held back a second. The timers were living one second in the future.
Now think about a program that sets a timer for 200 ms from now. To the confused timer system, "200 ms from now" is already 800 ms in the past. So the timer fires instantly. The program, which wanted to wait, sets the timer again. It fires instantly again. And again, and again.
%% title: The 2012 livelock (simplified) flowchart TD LS["Leap second inserted<br/>system clock set back 1 s"]:::hot --> MISS["Kernel skips clock_was_set()<br/>timers not told"]:::bad MISS --> AHEAD["Timer system now runs<br/>1 second in the future"]:::bad AHEAD --> SET["Program sets a short timer<br/>wake me in 200 ms"]:::service SET --> FIRE["Timer looks overdue<br/>fires immediately"]:::bad FIRE --> SET FIRE --> CPU(["CPU at 100%<br/>busy doing nothing"]):::bad
This is called a livelock. The machine isn't crashed or frozen. It's running flat out, doing nothing useful. Programs that wait using lots of short timeouts, such as Java applications and MySQL, were hit hardest. That's why the first reports blamed Java and Cassandra, the Java-based database Reddit used. Jonathan Ellis, CTO of DataStax, the main company behind Cassandra, then pointed out the real culprit: "a kind of livelock in the Linux system calls responsible for timers."
Who got hit#
Reddit was offline for 30 to 40 minutes. At Mozilla, servers running Zimbra (email), Elasticsearch, HBase for crash reports and some MySQL databases slowed to a crawl, though Mozilla's notes say downtime for any single service was minimal. (A few Mozilla IT staff found their home Minecraft servers had the same problem.) In Australia, the Amadeus Altea reservation system used by Qantas and Virgin Australia was out for about an hour, and more than 400 Qantas flights were delayed. LinkedIn, Gawker, Foursquare and Yelp also reported trouble.
The fix: change the time to the same time#
Restarting the stuck program didn't help. Mozilla noticed this straight away: restarting MySQL or Java did not fix it. Rebooting the machine did, and so did something that sounds like a joke. You just set the clock to what it already says:
# The core line of Mozilla IT's quick fix, rolled out with Puppet on 30 June 2012
/etc/init.d/ntpd stop; date -s "`date`"Setting the date is a system-time change, and system-time changes make the kernel update its timer offsets. The timers stopped living in the future, and the CPU calmed down. John Stultz's proper kernel fix put the missing clock_was_set() call back into the leap-second code.
2017: The Day Time Went Backwards at Cloudflare#
The next leap seconds came on 30 June 2015 and 31 December 2016. The 2016 one landed on a company that runs a huge share of the internet's DNS.
DNS, the Domain Name System, turns names like example.com into IP addresses. Cloudflare answers DNS queries for its customers' domains, using its own DNS server called RRDNS, written in Go. Some customers point their domain at another name instead of an address (a CNAME record), so RRDNS sometimes has to look up that other name itself, using a pool of internal resolvers.
To pick the best resolver, RRDNS measured how fast each one answered. Here's the real code, from Cloudflare's postmortem:
// Update upstream sRTT on UDP queries, penalize it if it fails
if !start.IsZero() {
rtt := time.Now().Sub(start)
if success && rcode != dns.RcodeServerFailure {
s.updateRTT(rtt)
} else {
// The penalty should be a multiple of actual timeout
// as we don't know when the good message was supposed to arrive,
// but it should not put server to backoff instantly
s.updateRTT(TimeoutPenalty * s.timeout)
}
}Look at rtt := time.Now().Sub(start). "Now, minus when we started." In the Go of that time, time.Now() read only the wall clock. Cloudflare's resolvers were so fast that they normally answered in a few milliseconds. So if a lookup was in flight exactly when the clock was set back a second, rtt came out negative.
One negative reading wasn't enough to break things, because RRDNS smoothed its measurements (averaged them over many readings). But after a few bad readings, the smoothed value went negative too. Then that value was handed to Go's random number function, rand.Int63n(), to make a weighted choice between resolvers. rand.Int63n panics (crashes) if you give it a negative number.
%% title: How a negative second became failed DNS lookups flowchart TD LS["Leap second · wall clock<br/>set back 1 s"]:::hot --> RTT["rtt := time.Now().Sub(start)<br/>comes out negative"]:::bad RTT --> SM["Smoothed average of<br/>resolver speed turns negative"]:::bad SM --> PICK["Weighted resolver choice<br/>rand.Int63n(negative)"]:::core PICK --> PANIC["Go panics"]:::bad PANIC --> REC["Caught with recover()<br/>server keeps running"]:::service REC --> FAIL(["Some CNAME lookups fail<br/>for Cloudflare customers"]):::bad
Cloudflare's own summary of the root cause is worth remembering word for word: "the belief that time cannot go backwards."
- 00:00 — impact starts as the leap second lands
- 00:10 — escalated to engineers
- 00:34 — issue confirmed
- 00:55 — fix deployed to one canary server and confirmed
- 01:03 — fix confirmed in a canary data centre
- 01:23 — fix deployed in the most affected data centre
- 01:48 — fix being deployed everywhere
- 02:50 — fix rolled out to most affected data centres
- 06:45 — impact ends
- 0.2% — of DNS queries to Cloudflare affected at peak
- < 1% — of HTTP requests to Cloudflare hit an error
- 102 — Cloudflare data centres at the time; only a small number of machines were affected
- 90 min — to patch the most affected machines
- 6 h 45 min — from the first impact to the last
The fix had two parts. First, never record a negative measurement: if time skipped backwards, RRDNS would forget its current numbers and let them settle again. Second, restart the RRDNS servers to clear the bad values.
And Go itself changed. In Russ Cox's design document for monotonic time, Cloudflare's outage is cited by name: "A Go program measuring a negative elapsed time during a leap second caused CloudFlare's recent DNS outage." Since Go 1.9 (August 2017), every time.Time from time.Now() quietly carries a monotonic reading too, so time.Since(start) and t.Sub(start) are safe against wall-clock jumps.
What Good Looks Like: Subtracting Timestamps Safely#
Both stories are the same bug in different clothes. Here's what it looks like in everyday code, and the fix. The snippets below are illustrative.
Breaks when the wall clock jumps
start = time.time() # wall clock
while not job.done():
if time.time() - start > 30:
raise TimeoutError()
time.sleep(0.5)Clock set back → the timeout comes late. Clock jumps forward → a healthy job "times out".
Safe
start = time.monotonic() # only moves forward
while not job.done():
if time.monotonic() - start > 30:
raise TimeoutError()
time.sleep(0.5)Elapsed time can't be negative, whatever NTP or a leap second does.
One subtlety: a monotonic reading only makes sense on the machine that took it, and two machines' wall clocks are never perfectly in sync. So when you compare times from two servers, allow for skew and never assume the result can't be negative. And whatever clock you use, if a computed duration comes out negative, clamp it or throw it away before it reaches anything that might crash, exactly as Cloudflare's patch did.
%% title: Which clock should I use?
flowchart TD
Q{"What are you doing<br/>with the time?"}:::hot
Q -->|"measuring how long<br/>something took"| M["Monotonic clock<br/>time.monotonic · nanoTime · performance.now"]:::ok
Q -->|"showing or storing<br/>when something happened"| W["Wall clock in UTC"]:::service
Q -->|"comparing times from<br/>two different machines"| X["Neither alone<br/>allow for skew · expect negatives"]:::queue
M --> G(["Safe from leap seconds<br/>and clock corrections"]):::ok
W --> C(["Fine for dates · never<br/>subtract for durations"]):::noteSmearing the Second Away#
Some companies decided the safest leap second is one your servers never see. Instead of inserting 23:59:60 all at once, they smear it: for many hours around the leap, their time servers run the clock very slightly slow, so the extra second is absorbed gradually. No jump, no repeated second, no negative durations. The price is that, during the window, smeared clocks drift away from official UTC: with Google's centred smear, by up to about half a second.
Google has done this since 2008. Its recommended standard, which it says Amazon also uses in AWS, is a 24-hour linear smear from noon to noon UTC.
- 12:00 UTC the day before — time servers start running clocks very slightly slow
- Just before the leap — smeared time is just under half a second behind official UTC
- The leap second — UTC inserts 23:59:60; smeared time keeps ticking smoothly and is now just under half a second ahead
- 12:00 UTC the next day — the extra second is fully absorbed; smeared time and UTC agree again
- One extra second spread across one day: 1 s ÷ 86,400 s = 0.0000116
- That's about 11.6 parts per million, or each smeared second being about 11.6 µs longer than a real one
- Google notes this is well under NTP's 500 ppm maximum, and within the normal error of a machine's quartz clock
The catch: there's no single standard. Meta has described a 17-hour smear starting at 00:00 UTC; Bloomberg smeared over 2,000 seconds after the leap; Google itself used 20-hour smears in 2012, 2015 and 2016. Mix servers that smear differently (or not at all) and their clocks can disagree by up to about a second.
Retiring the Leap Second#
By 2022, the people who run the world's clocks agreed the cure had become worse than the disease.
On 18 November 2022, the General Conference on Weights and Measures (CGPM), meeting outside Paris, adopted Resolution 4. Its key line "decides that the maximum value for the difference (UT1-UTC) will be increased in, or before, 2035." In plain words: from 2035 at the latest, UTC will be allowed to drift further from the Earth's spin than 0.9 seconds, so leap seconds stop being added.
The resolution's reasons read like a summary of this case file. Leap seconds "create discontinuities that risk causing serious malfunctions in critical digital infrastructure", from satellite navigation to telecoms and power grids. Companies handle them in different, uncoordinated ways. And the Earth's recent behaviour raises "the possible need for the first negative leap second whose insertion has never been foreseen or tested." A negative leap second would skip 23:59:59 entirely, and Meta's engineers warned it "could have a devastating effect on the software relying on timers or schedulers."
According to Nature's report on the vote, the CGPM proposed that no leap second be added for at least a century, letting UTC and Earth time drift apart by about a minute. Russia voted against, wanting to push the change back to 2040 or later because of technical issues with its GLONASS satellite navigation system.
The Numbers#
- 27 — leap seconds added to UTC since 1972, the last on 31 December 2016
- 30–40 min — Reddit offline in 2012
- 400+ — Qantas flights delayed after the Amadeus outage
- −990 ms — what a 10 ms measurement becomes across a repeated second
- 11.6 ppm — how slow Google's 24-hour smear runs the clock
- 2035 — the latest date for the change that ends new leap seconds
Lessons for Any Engineer#
- Time is not a number that only goes up. The wall clock can jump either way. Any code that assumes "later minus earlier is positive" has a hidden bug.
- Two clocks, two jobs. Wall clock for "when", monotonic clock for "how long". Most time bugs are using the wrong one.
- Rare code paths rot. The 2012 kernel bug sat unnoticed for years because leap-second code almost never ran. If something happens once every few years, test it on purpose.
- One time source for the whole fleet. If you smear, every server must smear the same way; mixing sources means clocks that disagree.
- Restarting the app isn't always the fix. In 2012 the bad state lived in the kernel; only setting the time or rebooting cleared it. Know which layer holds the state.
- Defensive checks are cheap. Refusing to record a negative
rttis exactly how Cloudflare stopped its resolver picker crashing.
Back in Australia, airline staff worked through the queues by hand, and passengers eventually got where they were going, late. The world had needed exactly one more second, and a lot of software had never been tested with one.
Sources#
- Read
Cloudflare: How and why the leap second affected Cloudflare DNS (1 January 2017)
- Docs
Jonathan Corbet, LWN.net: Leaping seconds and looping servers (2 July 2012)
- Read
Linux kernel mailing list: ntp: Fix leap-second hrtimer livelock and Fix for leapsecond caused hrtimer/futex issue (July 2012)
- Read
Mozilla IT: MySQL and the Leap Second, High CPU and the Fix (30 June 2012, archived) and Mozilla's weekly status notes
- Docs
Rich Miller, Data Center Knowledge: Leap Second Bug Snarls Air Travel in Australia (1 July 2012)
- Docs
Robert N. Charette, IEEE Spectrum: Leap Second Causes Hiccups at Some Websites (2 July 2012)
- Read
Joab Jackson, PCWorld: Linux is culprit in leap-second lapses, Cassandra exec says (2 July 2012)
- Docs
Google: Leap Smear (Google Public NTP documentation)
- Read
Russ Cox: Proposal: Monotonic Elapsed Time Measurements in Go (2017) and the Go 1.9 release notes
- Blog
Meta Engineering: It's time to leave the leap second in the past (25 July 2022)
- Read
BIPM: Resolution 4 of the 27th CGPM (2022): On the use and future development of UTC
- Book
Elizabeth Gibney, Nature (republished by Scientific American): The Leap Second's Time Is Up: World Votes to Stop Pausing Clocks (18 November 2022)
- Read
NIST: Leap seconds
- Docs
IETF: RFC 5905, Network Time Protocol Version 4 (leap indicator)
- Read
Sign in to keep reading.
The whole course is free right now. Sign in to read the rest of this page and unlock the AI tutor that remembers you, progress that syncs, notes, mock interviews and running code.
Read it through to close the case.
0/11 sections read. Each one counts after a few minutes of actual reading in it; skipping ahead doesn't. The case closes itself once every section is read.
All case files →
Discussion