The Day Facebook Disappeared: How One Command Erased Three Apps From the Internet's Map

The Evening the Grey Ticks Stopped Turning Blue#

It's a Monday night in India, a little after 9 PM. You send your mother a WhatsApp message to say you've reached home. One grey tick. You wait. Still one grey tick.

You open Instagram to kill time. The feed won't refresh. You try Facebook. "Something went wrong." You switch Wi-Fi off and on, restart the phone, and finally do what everyone does: open another app to ask, is it just me?

It isn't just you. At that moment, on 4 October 2021, Facebook, Instagram and WhatsApp have vanished for billions of people at once. (The company was still called Facebook then; it renamed itself Meta three weeks later.) They will stay gone for about six hours.

Here is the strange part, the part this case file is about. Facebook's servers were not hacked. They were not on fire. Most of them were switched on and running the whole time. What disappeared was something far more basic: the internet's knowledge of how to reach them. For six hours, Facebook was a house with a perfectly good front door, but someone had erased its address from every map on Earth.

And when Facebook's own engineers tried to fix it, they ran into a second problem. The tools they normally use to repair the network ran over the network. According to reporting at the time in The New York Times, some employees couldn't even badge into buildings and conference rooms.

To understand how one maintenance command did all of that, you only need two ideas: the internet's address book, and the internet's map.

Two Ideas First: the Address Book and the Map#

Every computer on the internet is reached by a number called an IP address, something like 157.240.1.35. Humans don't remember numbers, so we type names like facebook.com. Two systems turn that name into a working connection.

DNS: the internet's address book#

DNS (the Domain Name System) turns a name into an IP address. When your phone opens WhatsApp, it asks a DNS resolver (a lookup service run by your mobile operator, or a public one like Cloudflare's 1.1.1.1 or Google's 8.8.8.8) "what's the address for this name?"

If the resolver doesn't already have a recent answer saved, it asks the authoritative nameservers: the servers owned by the company itself that hold the official answer. For facebook.com, those were Facebook's own servers. Saved answers expire after a while (this lifetime is called the TTL, time to live), so resolvers go back to the authoritative nameservers regularly.

%% title: A normal DNS lookup (illustrative addresses)
sequenceDiagram
  autonumber
  participant P as Your phone
  participant R as DNS resolver
  participant N as Facebook nameservers
  participant F as Facebook servers
  P->>R: What is the address of facebook.com?
  alt answer saved and TTL not expired
    R-->>P: Here it is, from my saved copy
  else no saved answer
    R->>N: What is the address of facebook.com?
    N-->>R: 157.240.x.x
    R-->>P: 157.240.x.x
  end
  P->>F: Connect to 157.240.x.x

If the resolver can't reach the authoritative nameservers at all, it gives up and returns an error called SERVFAIL ("server failure"). Your app sees that as "this name doesn't work right now."

BGP: the internet's map#

DNS gives you an address. But how does a packet from your phone in Pune actually find its way to a building in another country? That's the job of BGP (Border Gateway Protocol).

The internet isn't one network. It's tens of thousands of separate networks (your mobile operator, a university, Cloudflare, Facebook) joined together. Each one is called an autonomous system (AS) and has its own number. Facebook's is AS32934; Cloudflare's is AS13335.

Each network uses BGP to tell its neighbours: "these blocks of addresses live with me, send their traffic my way." That's called announcing a route. A block of addresses is called a prefix, written like 185.89.218.0/23. Neighbours pass the news on to their neighbours, and soon every big router on the internet has an entry for that block in its table. When a network stops announcing a block, it's called a withdrawal, and routers delete the entry.

%% title: How a network tells the internet where it lives (simplified)
flowchart TD
  FB["Facebook · AS32934<br/>announces 185.89.218.0/23"]:::core
  T1["Transit network A"]:::edge
  T2["Transit network B"]:::edge
  ISP["Your mobile operator"]:::edge
  RT[("Router table<br/>185.89.218.0/23 → via A")]:::store
  P(["Your phone"]):::client
  FB -->|"BGP announce"| T1
  FB -->|"BGP announce"| T2
  T1 -->|"passes it on"| ISP
  T2 -->|"passes it on"| ISP
  ISP --> RT
  P -->|"packet for 185.89.218.x"| ISP

Here's the key thing to hold on to. If nobody announces a prefix, routers simply don't know it exists. There is no "Facebook is down" error on the map. There's just a blank space where Facebook used to be.

01Chapter 1: A Routine Maintenance Job#

Facebook runs its own private network, the backbone: in the company's words, "tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers."

Its buildings come in two kinds. There are huge data centres that hold millions of machines, where your messages and photos are actually stored and processed. And there are smaller edge facilities that connect the backbone to the rest of the internet, close to where people are. When you open the app, your request goes to the nearest edge facility, which talks over the backbone to a big data centre and brings the answer back.

%% title: How a request reached Facebook in 2021 (simplified)
flowchart TD
  P(["Your phone"]):::client --> ISP["Your mobile operator"]:::edge
  ISP --> EDGE["Nearest Facebook edge facility<br/>also answers DNS queries"]:::edge
  EDGE ==>|"private backbone"| DC1[("Large data centre")]:::store
  EDGE ==>|"private backbone"| DC2[("Large data centre")]:::store
  BB["Backbone routers<br/>tens of thousands of miles of fibre"]:::core
  EDGE --- BB
  BB --- DC1
  BB --- DC2

Engineers take bits of the backbone offline all the time: to repair a fibre, add capacity, or update router software. That's normal work.

On 4 October, during one of these routine jobs, a command was issued whose purpose was to assess the availability of global backbone capacity. In Meta's own words, it "unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally."

There was a safety net for exactly this. Meta says its systems are designed to audit commands like this one and stop the dangerous ones before they run. But "a bug in that audit tool prevented it from properly stopping the command."

Meta has not published the command itself, which routers it touched first, or what the audit tool's bug was. So we'll stop there and not guess. What matters is the result: every building was cut off from every other building, all at once.

02Chapter 2: The Safety Feature That Turned Off the Lights#

Losing the backbone was bad. What happened next is what made Facebook vanish from the whole internet.

Remember the small edge facilities? One of their jobs was answering DNS queries: they ran Facebook's authoritative nameservers. Those nameservers sit at well-known IP addresses, and those addresses are announced to the internet with BGP, like any other prefix.

The DNS servers had a sensible-sounding rule. Meta describes it like this: they "disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection."

Think about why you'd want that. If one edge facility's link to the data centres breaks, it can't give good answers. Better for it to quietly withdraw its announcement, so the internet's routers send DNS queries to a different, healthy facility instead. That's a health check: a server checking itself and stepping aside when it's sick.

%% title: The DNS server's self health check
flowchart TD
  S["Facebook DNS server<br/>at an edge facility"]:::core --> Q{"Can I reach<br/>the data centres?"}:::hot
  Q -->|"yes"| A["Keep announcing my prefix<br/>with BGP"]:::ok
  Q -->|"no · looks like my link is broken"| W["Withdraw my BGP announcement<br/>let another site answer"]:::bad
  A --> I(["Internet can reach this DNS server"]):::ok
  W --> G(["Internet forgets this DNS server"]):::bad

The rule assumes the problem is local: my link is broken, the others are fine. On 4 October the problem wasn't local. The entire backbone was gone. So every edge facility asked the same question, got the same answer, and stepped aside at the same moment.

Meta's summary is the most important sentence in the whole postmortem: "our DNS servers became unreachable even though they were still operational."

From outside, Cloudflare could see exactly this. Its engineers checked a public BGP route server for Facebook's DNS prefixes and got Network not in table for both 185.89.218.0/23 and 129.134.30.0/23. Interestingly, some of Facebook's other address blocks were still on the map. It didn't help: without DNS, nobody could find out which addresses to use.

%% title: The chain of failure
flowchart TD
  C["Maintenance command<br/>to assess backbone capacity"]:::hot --> AU{"Audit tool<br/>should block it"}
  AU -->|"bug · command runs"| BB["All backbone connections down<br/>data centres cut off"]:::bad
  BB --> HC["Every DNS server fails its<br/>health check at the same time"]:::bad
  HC --> WD["Every DNS server withdraws<br/>its BGP announcement"]:::bad
  WD --> SF["Resolvers worldwide get no answer<br/>SERVFAIL for facebook.com"]:::bad
  SF --> RS["Apps and people retry<br/>30x more DNS queries"]:::queue
  BB --> TL["Internal tools that need DNS<br/>and the network stop working"]:::bad
  TL --> SL(["Slow, hands-on recovery"]):::note

03Chapter 3: The Internet Notices#

Cloudflare runs one of the world's biggest public DNS resolvers, 1.1.1.1, and it watches BGP updates from across the internet. Its blog post that night is the best outside view of what happened. All times below are UTC, with India time (UTC + 5:30) in brackets.

Why the DNS traffic exploded#

When an app gets an error, it rarely gives up. It retries, often quickly. And when people get an error, they refresh, force-close the app, and open it again. Cloudflare saw this turn into "a tsunami of additional DNS traffic": resolvers handling 30 times more queries than usual.

Facebook was so big that its absence put load on everyone else. Cloudflare reports that 1.1.1.1 coped, with most lookups still answered in under 10 ms. But a dead service at that scale is a noisy neighbour.

04Chapter 4: Locked Out of Your Own House#

Inside Facebook, engineers now faced a nasty puzzle. Meta named two obstacles.

First, they couldn't get in remotely. Normally engineers reach routers and servers over the network. The network was the thing that was down. Big companies keep a backup path for this, an out-of-band network: a separate back door, used only when the main network is broken. Meta says plainly: "Our primary and out-of-band network access was down."

Second, their tools were broken too. "The total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this." Dashboards, chat, runbooks, deploy systems: if a tool finds other systems by name, and names don't resolve, the tool is blind.

%% title: The tools to fix the network needed the network
flowchart TD
  BB["Backbone down"]:::hot --> DNS["Facebook DNS unreachable"]:::bad
  BB --> RA["Remote access to routers"]:::bad
  BB --> OOB["Out-of-band access<br/>also down"]:::bad
  DNS --> T["Internal tools that look<br/>things up by name"]:::bad
  RA --> X(["Engineers cannot reach<br/>the routers to undo the change"]):::bad
  OOB --> X
  T --> X
  X --> ON["Send people to the<br/>data centres in person"]:::core

So Facebook did the only thing left: it sent engineers to the data centres in person. That took time too, and this time it was by design. In Meta's words, these facilities "are hard to get into, and once you're inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them." Secure access procedures had to be activated before anyone could work on the machines. Only then could the team confirm the problem and bring the backbone back.

On top of this, The New York Times reported that employees couldn't get into buildings and conference rooms with their security badges that day. Meta's posts don't mention badges or explain why, so treat this as press reporting, not a confirmed cause.

05Chapter 5: Turning the Lights Back On, Slowly#

Once the backbone was back, the DNS servers could reach the data centres again, passed their health checks, and announced their prefixes. The internet's map filled back in. You might think the job was done.

It wasn't. Six hours of silence had changed the physical state of the data centres. Meta says individual data centres were "reporting dips in power usage in the range of tens of megawatts." Switching every service back on at once would reverse that dip in one jolt, and that "could put everything from electrical systems to caches at risk."

Think of a city after a blackout. If every fridge, AC and water pump switches on in the same second, the grid can trip again. Caches are similar: they had gone cold, so if all of the world's users arrived at once, every request would miss the cache and land on the databases behind it.

So the team brought things back in stages. They could do this calmly because they'd practised. Meta has long run "storm" drills: exercises where it deliberately takes a service, a data centre or an entire region offline to test how everything copes. It had never simulated losing the entire global backbone, but the habits carried over.

In the end, Meta says, "our services came back up relatively quickly without any further systemwide failures." It also confirmed there was no malicious activity behind the outage, and that it had no evidence user data was compromised.

What Good Looks Like#

Meta hasn't published a list of the specific fixes it made after the review, so the patterns below are general engineering practice that this incident illustrates, not a description of Facebook's internal changes.

%% title: Illustrative: a health check with a safety floor
flowchart TD
  H{"Can I reach the data centres?"}:::hot
  H -->|"yes"| K["Keep announcing"]:::ok
  H -->|"no"| M{"Are most other sites<br/>failing too?"}
  M -->|"no · it's just me"| W["Withdraw · let others answer"]:::service
  M -->|"yes · shared failure"| F["Keep announcing<br/>and page a human"]:::ok
  W --> L{"Would this drop below<br/>the minimum number of sites?"}
  L -->|"yes"| F
  L -->|"no"| D(["Withdraw safely"]):::ok

The Numbers#

Lessons for Any Engineer#

• A server that is up but unreachable is down. Facebook's DNS servers were running the whole time. Measure availability from your users' side, not from the server's.
• Health checks can fail together. A rule that makes one sick node step aside can make every node step aside when they share a dependency. Ask "what if all of them see this at once?"
• Your repair tools must not depend on the thing they repair. Out-of-band access, admin tools and runbooks need their own network, their own name lookup and their own login.
• Guard rails need tests too. The command was dangerous; the audit tool was supposed to catch it. A safety check that fails silently is worse than none, because people trust it.
• Security and recovery speed pull against each other. Hard-to-enter buildings and hard-to-modify routers protect you every day and slow you down on the worst day. Choose that trade on purpose.
• Recovery is its own risk. After a long outage, cold caches, pent-up users and sudden power swings can cause a second outage. Bring traffic back in steps.

Somewhere around 2:50 AM in India, your mother's phone finally buzzed with your "reached home" message, hours late. Every server involved had been switched on the whole time. For about six hours, the internet just didn't know where to find them.

Sources#

• Blog

  Santosh Janardhan, Meta Engineering: More details about the October 4 outage (5 October 2021)

• Blog

  Meta Engineering: Update about the October 4th outage (4 October 2021, updated 5 October)

• Read

  Cloudflare: Understanding how Facebook disappeared from the Internet (4 October 2021)

• Read

  CNBC: Facebook shares drop nearly 5% after major site outage and whistleblower interview (4 October 2021)

• Read

  TASS, quoting Downdetector: Facebook outage named largest in history (October 2021)

• Book

  Reporting by The New York Times on employees' badge access, as summarised in 2021 Facebook outage (Wikipedia, citing NYT)