Many Cloudflare features rewrite web pages as they pass through its servers, for example hiding email addresses from spam bots. To do that, Cloudflare's servers parse the HTML of customers' pages, many sites sharing the same machines.
On 18 February 2017 at 00:11 UTC, Tavis Ormandy of Google's Project Zero tweeted asking for a Cloudflare contact. He had been seeing corrupted pages served by Cloudflare. In some unusual cases, Cloudflare's servers were running past the end of a buffer and sending back whatever sat in memory next to it, which could hold "HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data". Some of it had been cached by search engines.
The old parser was built with Ragel, a tool that generates C code. The generated code held this check:
if ( ++p == pe ) goto _test_eof;
It checked whether a pointer had reached the end of the buffer with ==. A missing step in
Cloudflare's own Ragel code (its bug, Cloudflare stressed, not Ragel's) let the pointer jump past
the end, and the check never fired. With >=, "jumping over the buffer end would have been caught".
The twist: that old parser had carried the bug for years, harmlessly. Cloudflare was replacing it with a new one, cf-html. The new parser changed how buffers were passed around, and that woke the old bug. It only leaked when a page ended in broken HTML and certain features were on.
The details reached Cloudflare at 00:32 UTC. It turned off email obfuscation worldwide at 01:19, 47 minutes later, and was "completely finished globally in under 7 hours". Then the cleanup: with Google, Yahoo, Bing and others it found and purged 770 cached pages of leaked memory, across 161 domains. Customer SSL private keys were not leaked: SSL ran in a separate, isolated process.
- Leaks possible from 22 September 2016; worst from 13 to 18 February 2017
- About 1 in every 3,300,000 requests could leak at the peak (about 0.00003%)
- 47 minutes to the first mitigation, under 7 hours to the full fix
- 770 cached URIs across 161 domains purged from search engines
Why it matters: On shared machines, one memory bug can hand out whatever else is in memory. Use
safe bounds checks (>=, not ==), fuzz old code before you touch what surrounds it, and keep the most
sensitive secrets in a separate, isolated process.