Three months, said the tool. The rewrite said nine days.

transcript

Discord moved trillions of messages to a new database. The first tool said three months. The rewrite said nine days.

Early 2022. Discord's messages sit on 177 Cassandra servers. Each channel's messages live together. One channel gets busy. All its reads hit the same few machines. They slow down. And so does every other request routed there.

No panic, though. By 2020, every other database was on ScyllaDB. Then a shield in Rust. Many people, one message? One query. The first plan? New messages to ScyllaDB first. Copy the history behind them.

The first copying tool estimated three months. So they rewrote it in Rust. Their words: meme-driven engineering. Up to 3.2 million messages a second. The new estimate? Nine days. Fast enough to drop that plan, and switch everything at once.

Several days later: 99.9999 percent. And there, it got stuck. Tombstones. Markers for deleted data Cassandra never cleaned up. They cleared them. May 2022, they switched over. With cake.

Before: 177 servers. Old-message reads, up to 125 milliseconds at p99. After: 72 servers. Reads at 15 milliseconds. Writes? A steady 5 milliseconds.

Migrations that look sudden are usually the last step of a plan. Small databases first. A shield in front. Then a copy fast enough to switch everything at once.

Full breakdown in the lecture.