On the weekend of 3 and 4 February 2018, Fortnite hit a new peak of 3.4 million concurrent players, "far beyond our wildest growth expectations", in Epic Games' words. The load caused 6 different incidents over Saturday and Sunday, some partial, some total.
One of them hit Epic's Account Service, which signs players in and checks their access tokens. In front of its Java application sat an Nginx proxy with one shortcut: token checks were answered from a Memcached cache, and all other calls simply passed through to the application.
On Sunday, from 18:30 to 20:20 UTC, Memcached started failing under the load, its network and connections saturated. Nginx got stuck waiting on each cache call until its 100 ms timeout, and soon ran out of free worker threads. Now it couldn't serve any traffic, including the load balancer's health checks. Seeing the missed checks, the load balancer pulled all the nodes out of rotation: full downtime. The Java application had its own protection against overload, but it didn't help, because the traffic couldn't get through to it.
Players already in a match were only moderately affected. But signing in and out was mostly blocked, and the Epic Games Launcher signed players out with an error.
Epic's next steps: send all traffic except token checks straight to the Java application, increase Memcached capacity significantly, and then remove the Nginx and Memcached pair altogether.
- 3.4 million concurrent players at the peak
- 6 incidents in two days
- 100 ms: the cache timeout that tied up every Nginx worker
- 18:30 to 20:20 UTC: the Account Service outage on Sunday
- 124,000 requests a second to the game's profile service at peak, and 318,000 database reads a second
Why it matters: A helper in front of your service can take the whole fleet down with it. If a slow dependency can tie up every worker, the health checks fail too, and the load balancer finishes the job. Keep shortcuts off the path of traffic that doesn't need them, and give them tight timeouts.