The $440M Deploy: Knight Capital's 45 Minutes

Understanding the Problem: The Morning the Router Wouldn't Stop#

It's Wednesday, August 1, 2012, in Jersey City, New Jersey. The US stock market opens at 9:30 a.m. Eastern, the same as every weekday.

You work at Knight Capital, one of the biggest names in American trading. On a normal day, Knight handles roughly one in every ten shares traded in US-listed stocks, often quietly filling orders that retail investors place through their brokers.

Today is a launch day. The New York Stock Exchange is starting a new program for retail orders, and Knight's new code for it started going out to the servers last week.

At 9:30 the bell rings. Within minutes, something is badly wrong.

Knight's order router, the system that sends orders out to the exchanges, starts firing orders into the market far faster than any customer asked for, in more than a hundred stocks. Prices lurch. Positions pile up in an internal account, and nobody watching can tell where they're coming from.

Nobody pulls the plug. For about 45 minutes, the machine keeps going.

By the time it stops, Knight has bought and sold its way into a position worth billions of dollars that it never wanted. Selling its way back out costs about $440 million (Knight's figure the next day; the SEC later put the final loss at more than $460 million).

That's more than Knight's entire revenue for the previous quarter.

The cause wasn't a hacker, a rogue trader, or a market crash. It was a deploy that reached seven servers out of eight, a feature flag that was reused for a new job, and a piece of dead code that nobody had removed in nine years.

Using the facts in the US Securities and Exchange Commission's 2013 order against Knight, let's walk through how those three ordinary things lined up, and how to keep them from lining up in your systems.

The Anatomy of an Order Router#

Before we can break it, we need to understand what the system was supposed to do.

When a customer (say, a retail brokerage) sends Knight an order like "buy 1,000 shares", that's a parent order. Knight's router, SMARS, looks at where shares are available and splits it into smaller child orders, each sent to an exchange or trading venue.

As child orders fill, the router keeps a running count. When the count reaches 1,000, the parent order is complete, and the router stops sending children. That last part is the most important line in the whole design: stop when you're done.

%% title: What SMARS is supposed to do (example numbers)
flowchart TD
  C(["Customer order<br/>buy 1,000 shares"]):::client --> P["Parent order"]:::service
  P --> S["SMARS router<br/>splits into child orders"]:::core
  S --> V1["Exchange A<br/>child · 400 shares"]:::external
  S --> V2["Exchange B<br/>child · 350 shares"]:::external
  S --> V3["Exchange C<br/>child · 250 shares"]:::external
  V1 -.->|"filled"| Q{"Filled count<br/>reached 1,000?"}
  V2 -.->|"filled"| Q
  V3 -.->|"filled"| Q
  Q -->|"yes · stop sending"| D(["Parent complete"]):::ok
  Q -->|"not yet"| S

The SEC called SMARS an "automated, high speed, algorithmic router". On its own it handled about one percent or more of all US-listed stock trading. Speed is the point of a system like this, and also the danger: it can turn a small mistake into a huge one before a human has read the first alert.

Three Terms You Need#

Keep those three in mind. On August 1, all three failed at once.

01Chapter 1: A Feature Called Power Peg#

Years before 2012, SMARS had a feature called Power Peg. The SEC's order doesn't describe what Power Peg was for in trading terms, and we won't guess. What matters is this: Knight stopped using Power Peg in 2003.

Stopped using it. Didn't delete it.

The code stayed on the production servers, "present and callable", in the SEC's words. And it was switched on by a particular flag. If an order arrived with that flag set to "yes", SMARS would run it through Power Peg.

Then, in 2005, Knight changed the code around it. Power Peg had logic that counted how many shares of the parent order had been filled: the "stop when you're done" counter. Knight moved that counting function to an earlier point in the SMARS code.

Nobody re-tested Power Peg afterwards. Why would they? Nobody used it. So from 2005 on, the servers carried a sleeping feature that, if anything woke it, would send child orders without end, because the part that told it to stop had been moved out from under it.

%% title: Power Peg after the 2005 change (simplified, illustrative)
flowchart TD
  F(["Order arrives<br/>with the old flag set"]):::client --> PP["Power Peg code<br/>retired in 2003, never deleted"]:::hot
  PP --> SEND["Send another child order"]:::bad
  SEND --> X{"Has the parent<br/>been filled?"}
  X -.->|"the counter that answered this<br/>was moved away in 2005"| GONE["Nobody is asking"]:::note
  X -->|"never answered · loop again"| SEND

Here is the same idea as code. This is illustrative only. Knight's real code has never been published; this just shows the shape of the bug.

# Illustrative only - not Knight's code.
def power_peg(parent):
    while True:
        send_child_order(parent)
        # Before 2005 a check like this lived here:
        #   if filled_quantity(parent) >= parent.quantity:
        #       break
        # It was moved elsewhere, and this path was never re-tested.

A loop without its exit. Every engineer has written one by accident. This one was wired to the stock market.

02Chapter 2: A New Program, an Old Switch#

Fast forward to summer 2012. The NYSE is launching its Retail Liquidity Program (RLP) on August 1. To let its customers take part, Knight writes new code for SMARS.

Two design decisions in that work set up everything that followed.

1. The new RLP code would go where Power Peg's code used to be. Knight intended to delete Power Peg as part of this change.
2. The RLP code reused Power Peg's old flag. The same flag that used to mean "use Power Peg" would now mean "use RLP".

On paper, this is tidy: the old feature goes, the old switch gets a new job, no new field needed. But the plan quietly assumed every server would get the new code. On any server still running the old code, that flag still meant exactly what it meant in 2003.

if order.flag:
    route_via_rlp(order)

if order.flag:
    power_peg(order)

03Chapter 3: Seven Out of Eight#

SMARS ran on eight servers. Beginning on July 27, 2012, Knight deployed the new RLP code in stages, putting it on a limited number of servers on successive days.

Staging is usually wise. But according to the SEC, "one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers." That's the root event: one manual copy, missed on one machine. What made it fatal was what didn't happen next:

• Knight did not have a second technician review the deployment.
• Knight had no written procedures that required such a review.
• No one realised that the eighth server still had Power Peg on it and didn't have the RLP code.

The SEC put it bluntly: "a written procedure requiring a simple double-check of the deployment of the RLP code could have identified that a server had been missed and averted the events of August 1."

%% title: The fleet on the morning of August 1
flowchart TD
  U(["RLP-eligible orders<br/>flag set to yes"]):::client --> LB["Order flow into SMARS"]:::edge
  subgraph FLEET["SMARS · eight servers"]
    S1["Servers 1 to 7<br/>new RLP code"]:::ok
    S8["Server 8<br/>old code · Power Peg still there"]:::hot
  end
  LB --> S1
  LB --> S8
  S1 --> OK(["Orders handled correctly"]):::ok
  S8 --> BAD(["Child orders sent<br/>without stopping"]):::bad

Notice what's missing from this picture: any box that checks all eight servers are the same. The deploy was considered done because a person believed it was done.

04Chapter 4: 97 Emails Before the Bell#

On the morning of August 1, Knight received some RLP-eligible orders meant for pre-market trading (before the 9:30 open). SMARS processed them, and starting at about 8:01 a.m. Eastern, an internal Knight system began sending automated emails.

These emails were called "BNET rejects". They referenced SMARS and described the error as "Power Peg disabled."

By 9:30, Knight's system had sent 97 of them to a group of Knight staff.

Read that again. Power Peg disabled. A feature dead since 2003 was showing up in error messages on launch morning. That's about as loud as a clue gets.

But these emails weren't designed to be alerts, and the SEC found that "Knight personnel generally did not review them when they were received." They were caused by the failed deploy and, in the SEC's words, "provided Knight with a potential opportunity to identify and fix the coding issue prior to the market open." Nobody acted on them before the open, or used them to diagnose the problem afterwards.

The Incident Timeline: From 2003 to the Rescue#

The whole chain, start to finish. Times on August 1 are US Eastern.

05Chapter 5: The 45 Minutes#

At 9:30 the market opened, and the orders that had been waiting for the opening auction started flowing.

Seven servers handled their share of the RLP orders correctly. But orders that landed on the eighth server, carrying the repurposed flag, triggered Power Peg. And because the fill counter had been moved away in 2005, that server "continuously sent child orders, in rapid sequence, for each incoming parent order without regard to the number of share executions Knight had already received."

The cruel detail: another part of Knight's system did know the parent orders had been filled. That information never reached SMARS. One component knew the job was done; the one sending orders didn't.

Just 212 parent orders went through the broken path. Out came millions of child orders, and more than 4 million executions.

This wasn't only Knight's problem. Those trades moved real prices for everyone else. In 75 of the stocks, Knight's trades made up more than 20% of the volume and helped move prices by more than 5%. In 37 of them, Knight was more than half of all trading and prices moved more than 10%. Some investors on the other side got worse prices because of it, and some got better ones.

The New York Times summed it up in one line the next day: "$10 million a minute."

Why didn't something stop it?#

Knight did have tripwires. They just weren't in the right place.

The SEC also found SMARS had no control "to compare orders leaving SMARS with those that entered it" (212 in, millions out), and no procedure to halt SMARS in response to its own abnormal behaviour. In other words: no kill switch, and no rules for when to pull one.

06Chapter 6: Why the Rollback Made It Worse#

Put yourself in the war room. Something is very wrong, and the only recent big change is the new RLP code. The obvious move? Roll it back.

That's what Knight's team did. In one of its attempts to fix the problem, Knight uninstalled the new RLP code from the seven servers where it had been deployed correctly.

According to the SEC, "this action worsened the problem, causing additional incoming parent orders to activate the Power Peg code that was present on those servers, similar to what had already occurred on the eighth server."

Why? The rollback changed the code, not the orders. RLP orders kept arriving with the flag set to "yes", and on the rolled-back servers that flag once again meant "Power Peg". One broken server became several.

%% title: The rollback, step by step
sequenceDiagram
  autonumber
  participant UP as Upstream order flow
  participant S8 as Server 8 · old code
  participant S17 as Servers 1 to 7
  participant M as Market
  UP->>S8: RLP order · flag = yes
  S8->>M: child orders · no stop
  UP->>S17: RLP order · flag = yes
  S17->>M: handled correctly as RLP
  Note over S17: team uninstalls the new RLP code
  UP->>S17: next RLP order · flag still yes
  S17->>M: now Power Peg too · child orders · no stop
  Note over S8,M: the orders still carried the flag

This catches even experienced teams: a rollback restores code, not the world around it. Flags, configuration and traffic in flight stay as they were, and the old code may read them in its own, older way.

The SEC said it directly: Knight "needed clear guidance for its technology personnel as to when to disconnect a malfunctioning system from the market." Knight had no written incident response procedures for this. It relied mainly on its technology team to diagnose the problem live, while the system kept trading.

What Good Looks Like: Defence in Depth#

No single fix would have made Knight safe, and no single mistake caused this. It took a stack of small gaps lining up. The answer is a stack of small defences, any one of which would have broken the chain.

%% title: Layers that each break the chain
flowchart TD
  D["Deploy<br/>automated · same artifact everywhere"]:::service --> V{"Every server reports<br/>the expected version?"}
  V -->|"no"| STOP1(["Block the launch"]):::bad
  V -->|"yes"| FL["Fresh flag for the new feature<br/>old code deleted first"]:::core
  FL --> AL["Unexpected errors page a person<br/>before the open"]:::queue
  AL --> OUT{"Orders out far more<br/>than orders in?"}:::hot
  OUT -->|"yes"| KS(["Kill switch halts the router"]):::bad
  OUT -->|"no"| CAP{"Firm-wide capital<br/>threshold exceeded?"}
  CAP -->|"yes"| KS
  CAP -->|"no"| MKT(["Orders reach the market"]):::ok

Let's turn each layer into something you can actually do.

1. A deploy isn't done until the fleet says so#

Don't trust that a copy happened. Ask every machine what it's running. Each server reports a version number or build hash, and the deploy tool refuses to call the job finished until every one matches. That is the SEC's "simple double-check", done by a machine, every time.

2. The rest of the stack#

• Never give an old flag a new job. Flags are cheap; make a new one. With a fresh flag, a server that missed the deploy would simply have ignored RLP orders.
• Delete dead code on purpose, in its own release. If any input can reach it, it can run. Knight had no written protocol for re-testing unused code after changing it.
• Design alerts as alerts. An alert has an owner, a severity and a person who gets paged. An error naming a feature that shouldn't exist should stop a launch, not land in an inbox.
• Guard the last hop. Compare orders out with orders in (212 in, millions out is impossible on purpose), and wire a firm-wide capital limit that automatically blocks new orders once crossed. Those checks cap the damage whatever the bug is.
• Give someone permission to pull the kill switch. Write down who can halt the system and when, and practise it. A few minutes of lost business is nothing next to $10 million a minute.

It Wasn't the First Warning#

In October 2011, after a weekend disaster-recovery test, a Knight market-making desk kept using test data to generate live automated quotes on Monday morning. It cost nearly $7.5 million. The SEC found Knight fixed that exact problem but "did not broadly consider whether it had sufficient controls to prevent the entry of erroneous orders, regardless of the specific system." Ten months later, a different bug walked through the same gap.

That's the pattern behind many repeat outages: the review patches the exact bug, and the next incident comes through a different door in the same weak wall. The SEC brought its case under the Market Access Rule (Rule 15c3-5), adopted in 2010 and in force since 2011, which requires brokers to have controls that block erroneous orders and orders over pre-set capital limits: wall-level fixes, not door-level ones.

Lessons for Any Engineer#

You probably don't run a trading firm. But you almost certainly have servers that drift out of sync, flags someone wants to recycle, and old code nobody wants to touch.

• Partial deploys are a normal failure. "Seven out of eight" will happen to you; make launch impossible until the fleet is verified.
• Flags carry meaning, and meaning has versions. A reused flag ties a new feature to whatever old code still reads it.
• Rollback is not a reflex. Ask what it will restore, including what the old code thinks your current flags mean.
• Stop first, debug second. The investigation can wait a few minutes. The damage can't.
• Alerts need owners. 97 warnings were sent. None was acted on.

Summary: The Switch That Stayed Wired#

Here's the strangest thing about August 1, 2012: nothing that went wrong was exotic.

An old feature left in place. A loop that lost its exit. A flag with a second job. One server missed in a manual copy. Emails sent to an inbox. A rollback under pressure. Each happens at ordinary companies every week, mostly without consequence.

What made Knight different was speed and reach. A router wired to the US stock market turned everyday gaps into about $440 million of losses in less time than a lunch break. Within days the firm needed $400 million of emergency funding; within a year it was no longer independent.

So the next time someone wants to reuse an old flag, or says the deploy "should" be on every box, or mentions that weird error email everyone ignores, remember the eighth server. It was doing exactly what it had been told to do in 2003. Nobody had told it anything since.

Sources#

• Paper

  US Securities and Exchange Commission, In the Matter of Knight Capital Americas LLC, Exchange Act Release No. 70694, File No. 3-15570 (October 16, 2013): sec.gov/files/litigation/admin/2013/34-70694.pdf

• Docs

  Knight Capital Group, press release of August 2, 2012 (Form 8-K, Exhibit 99.1): sec.gov

• Docs

  Knight Capital Group, Form 8-K of August 6, 2012 ($400 million convertible preferred financing): sec.gov

• Docs

  Nathaniel Popper, "Knight Capital Says Trading Glitch Cost It $440 Million", The New York Times DealBook, August 2, 2012: archive.nytimes.com