Preface#
GOV.UK Notify sends text messages for UK central government, local authorities and the NHS: flood warnings, sign-in codes. It used two SMS providers. All traffic went through one; if it returned a single 5xx or its delivery receipts slowed, Notify swapped all traffic to the other automatically. The team rebalanced by hand "maybe once a week" (GOV.UK Notify, GDS blog, April 2020). Then, towards the end of 2019:
- A spike arrives
- "A large spike in requests to send text messages" goes to one provider, which "couldn't handle the load and started to fail"
- Failover works as designed
- Notify swaps all traffic to the second provider
- The second provider fails too
- "Sending a large amount of traffic out of nowhere caused them to start returning errors too"; the providers likely "needed time to scale up"
- The fix
- Every text now goes to a provider picked at random, roughly 50/50, and errors shift traffic 10 percentage points at a time
By the team's own account, the providers most likely just needed time to scale up: the platform's own traffic shape broke both. The failover was correct and still made things worse, because it moved the whole spike at once onto a provider that hadn't seen it coming.
Day 1 designed the platform, Day 2 its delivery pipeline, Day 3 the gateways and Day 4 presence and fan-out. Today we run it: what to measure and promise, how to survive campaigns, provider outages and lost regions, what it costs, and the edge cases of millions of users. Then we walk the whole week as one interview answer.
Day 5 runs the platform. GOV.UK Notify's all-or-nothing SMS failover broke both providers, so traffic is split and shifted in steps. Measure the delivery funnel (accepted is not delivered), latency from created_at and lane age; SLOs stop at the provider hand-off, and ~14 minutes of stall at peak spends a month's critical budget. Campaigns are paced jobs (~5,500/s from 10:07), lanes reach the provider through reserved quota and a separate OTP sender, breakers work per provider and route, and an APNs outage sends only critical alerts to SMS. SMS cost is messages × segments × price (~$11.9M to ~$5.9M a month); a lost region is fenced, replayed and re-admitted; and the week closes as one 45-minute interview.
Part I: Knowing What's Happening#
Chapter 1: What to Measure#
1.1 The Simple Definition#
Operating the platform means answering three questions at any moment, with numbers: are notifications arriving on time (per class: an OTP and a sale banner have different clocks), where are they stuck (our queues, a provider, the device), and what is it costing (SMS money, user patience, provider goodwill)?
1.2 The Delivery Funnel#
Count every notification at every stage, labelled by priority class, channel and provider, and the funnel shows where it's stuck.
%% title: One notification's funnel, and where each metric is measured
flowchart TD
A["Accepted by the API<br/>created_at stamped"]:::core --> B["Routed<br/>preferences, quiet hours, dedup"]:::service
B --> C@{ icon: "sd:queue", label: "Lane topic<br/>oldest message age" }
C --> D["Channel worker<br/>rate limiter, breaker"]:::service
D --> E@{ icon: "tech:apple", label: "APNs<br/>200 or an error" }
D --> F@{ icon: "tech:firebase", label: "FCM<br/>200 or an error" }
D --> G@{ icon: "sd:sms", label: "SMS provider<br/>queued, then a receipt" }
D --> H@{ icon: "sd:gateway", label: "Gateways<br/>client ack by seq" }
E -.->|"no receipt"| I@{ icon: "sd:phone", label: "Device<br/>shown, opened" }
F -.->|"Android data, days late"| I
G -.->|"delivery receipt"| I
H -->|"ack"| I
class C queue
class E,F,G external
class H edge
class I clientThe dotted edges are the hard part: "provider accepted" isn't "delivered", and the providers say so.
- APNs is "a best-effort service"; notifications "may also get throttled, saved in storage, and in some cases, not delivered" (Apple docs).
- FCM counts as sent anything "enqueued for delivery or ... successfully passed to a third-party service"; its delivery data covers Android only and is "delayed by up to 5 days" (Firebase docs).
- SMS has receipts: Twilio's
sentmeans "the nearest upstream carrier accepted the outbound message",deliveredthat it "received confirmation of outbound message delivery" (Twilio docs). - In-app is the one channel where we see the device: the client acks the sequence number (Day 3).
1.3 The Signals Worth a Graph#
| Signal | How to measure it | Why |
|---|---|---|
| End-to-end delivery latency | Provider accepted (or client ack) minus created_at, p50/p99 per class and channel |
The user's clock |
| Oldest message age per lane | Oldest unconsumed message in each priority topic | 2M in the bulk lane can be fine; 2,000 in critical isn't |
| Send success by channel and provider | Accepted ÷ attempted, split retryable (429, 5xx, timeouts) and permanent (bad token, bad number) | Retryable: the provider is unwell. Permanent: our data is |
| SMS receipt rate and latency | delivered ÷ sent, time to receipt, per provider and country |
200s can hide stalled carriers |
| Fan-out lag (Day 4) | Log append to last gateway publish; inbox pointer lag | Hot rooms show here first |
| Inbox sync backlog (Days 3–4) | Devices with cursors far behind; sync duration | Rises after reconnect waves |
| Connections (Day 3) | Sockets per gateway, reconnects per second, admission refusals, ack-timeout fallbacks | Storms show here first |
| Presence pushes per change (Day 4) | Presence events delivered ÷ state changes | Leaky subscriptions |
| Push token invalidation rate | APNs 410 and BadDeviceToken, FCM UNREGISTERED per 1,000 sends, by app version |
A jump after a release is a bug |
| Spend per channel per hour | Segments × price, by type and country | Money, and fraud |
Two rules keep these honest. Measure from created_at, not dequeue time: a worker timing "picked up to sent" reports 200 ms while messages wait 4 minutes. And keep labels small: ids go in logs and traces (Week 10, Day 2 · Observability).
# workers/metrics.py: record the user's clock, not the worker's (illustrative)
import time
from prometheus_client import Histogram, Counter
E2E = Histogram("notif_e2e_seconds", "created_at to provider accepted",
["priority", "channel", "provider"],
buckets=(0.5, 1, 2, 5, 10, 30, 60, 300, 900, 3600))
SENDS = Counter("notif_sends_total", "send attempts by outcome",
["priority", "channel", "provider", "outcome"]) # ok | retryable | permanent
def record(n, provider, outcome):
SENDS.labels(n.priority, n.channel, provider, outcome).inc()
if outcome == "ok":
E2E.labels(n.priority, n.channel, provider).observe(time.time() - n.created_at)Green dashboard, angry users
It's 19:05. Push success is 99.8% and errors are flat, but support is flooded: OTPs take four minutes. Which metric would have shown it, and why did the success rate stay green?
One strong answer
- Oldest message age in the critical lane, or p99 end-to-end latency for critical: either jumps within a minute.
- Success rates only count attempts. A message waiting in a topic hasn't failed, so a four-minute queue looks perfect until it times out.
- The usual cause: a campaign sharing the OTPs' lane, workers or provider quota (Chapter 3).
Chapter 2: SLOs for Delivery#
2.1 What to Promise#
Week 10, Day 1 · SLIs, SLOs and SLAs teaches the method. Here, an SLO must cover what we control and can measure; we can't see an iPhone show a banner, so push SLIs stop at the provider's acceptance. Day 1 set the targets: transactions in 1–2 seconds, security alerts near-instant, 99.9% for critical.
| Class | SLI | SLO (30 days) |
|---|---|---|
| Critical (OTPs, security) | Handed to a provider, or acked in-app, within 10 s of created_at |
99.9% |
| Transactional (payments) | Handed to a provider within 30 s | 99.5% |
| In-app, client connected | Client ack within 2 s of the inbox write | 99% |
| SMS receipts, critical | delivered within 2 minutes of sent, per provider |
98% |
| Bulk (campaigns, digests) | Sent within the campaign window, never in quiet hours | 99% of recipients |
Bulk has no availability target: pausing a campaign to protect OTPs is the system working. The SMS receipt SLO is partly outside our control, so it drives failover (Chapter 4).
How long can the critical lane stall?
Assume critical notifications (OTPs and security alerts) are 5% of Day 1's 500M a day; Day 1 doesn't split volume by class, so this share is illustrative. The SLO is 99.9% within 10 seconds over 30 days, and peak is 3× average. How many late critical notifications does a month allow, and how long a full stall at peak spends all of it?
One strong answer
- Volume: 25M a day × 30 = 750M. Budget: 0.1% = 750,000 late notifications.
- Peak: 25M ÷ 86,400 ≈ 290 a second, × 3 ≈ 870 a second.
- Stall: 750,000 ÷ 870 ≈ 860 s: about 14 minutes at peak spends the month.
- So one bad deploy or one shared provider queue can burn a month in a lunch break: the critical lane gets its own workers and quota, and a fast-burn page.
2.2 What Pages#
Page on symptoms users feel, with Week 10's multi-window burn-rate alerts (14.4× burn over an hour pages at once), plus the few causes that need a human within minutes:
| Alert | Severity | Why |
|---|---|---|
| Critical-lane SLO fast burn | Page | Users can't sign in |
| Breaker open on a critical-path provider | Page | Failover is running |
Any APNs ExpiredProviderToken (403) |
Page | Apple rejects provider tokens over an hour old: our signer died, every iOS push fails |
| Connections down 5% in a minute | Page | Day 3's reconnect storm |
| SMS spend 3× the same hour last week | Page | A retry loop or SMS pumping |
| Token invalidation 2× after a release | Ticket | A registration bug |
| Bulk lane past its window | Ticket | Marketing is late, nothing is broken |
- Lane age high? Our side: pause any campaign, then check critical workers and deploys.
- Provider slow or failing? 429 → slow down; 5xx or timeouts → check the breaker and failover.
- APNs 403s? Check the provider-token signing job and key first.
- Tell support what users see; update every 15 minutes.
Part II: Bursts and Provider Failures#
Chapter 3: Campaign Mode and Priority Lanes#
3.1 The Shape of a Campaign#
Day 1 allows 10M marketing notifications an hour on top of ~5,800 a second: only 2,800 a second more if spread evenly, but someone clicks Send and 10M rows land at once.
Bursts can be planned. During the 2024 Super Bowl, Duolingo pushed to 4 million learners, 95% within 3.9 seconds, after engineers had to "redesign the entire system to support the volume" (Duolingo, February 2024). A planned burst is an event; an unplanned one is an incident.
FCM says how it wants to be treated (Firebase docs):
- Default quota: 600,000 messages a minute per project (10,000 a second); above it,
429 RESOURCE_EXHAUSTED. - Ramp: "at minimum, ramp from 0 to the max RPS across a 60 second time-window."
- Avoid the quarter hours: FCM gets "more than double traffic during the first 30 seconds to 2 minutes of each hour", so stay out of "a 2 minute window of each of the :00, :15, :30, and :45 minute marks."
- Per Android device: 240 messages a minute and 5,000 an hour, and "don't routinely send messages near this maximum rate."
3.2 Campaign Mode#
A campaign is a job, not 10M API calls:
- Snapshot the audience from a replica or warehouse, never the primary.
- Filter once: consent, preferences, frequency caps, quiet hours. Uber's Consumer Communication Gateway schedules each user's pushes under limits like "at most 2 pushes per day" and "8 hours between pushes" (Uber Engineering, 2022); LinkedIn's Air Traffic Controller "will rate-limit upstream applications to prevent them from accidentally spamming members" (LinkedIn Engineering, 2018).
- Pace into the bulk lane: 10M over 30 minutes ≈ 5,500 a second, ramped, starting at 10:07, not 10:00.
- Wave by time zone (Chapter 7).
- Key each send
(campaign_id, user_id, channel)so a restarted job doesn't send twice. - Kill switch: pause, resume, cancel, checked before every batch.
3.3 Priority Lanes All the Way to the Provider#
Day 1's interviewer: "Transaction and security notifications should never be delayed by marketing campaigns." So: a topic per class (notifications.critical, notifications.transactional, notifications.bulk), a worker pool per lane scaled separately, and lag watched per lane.
In one shared queue, a 10M campaign that lands at once puts 9.2M messages ahead of Rohan's 20:00:40 OTP, which leaves at ~20:08 after its code has expired. With a topic and worker pool per class (notifications.critical, notifications.transactional, notifications.bulk), the campaign is paced into bulk at ~5,500 a second and nothing is ever in front of the OTP. The lanes still merge at the provider (FCM's 600,000 a minute per project, one queue per SMS sender), so token buckets reserve a share bulk can't touch (Day 2 assumes 4,000/s, leaving 6,000/s for bulk) and OTPs get their own SMS sender with a 120 s validity period.
Netflix's RENO "segments incoming events by priority and routes them to priority-specific AWS SQS queues and the corresponding compute instance clusters", so one can fail without the others (Netflix Technology Blog, 2022, via InfoQ). Uber's push platform writes to the socket "in descending order of priority" (Uber Engineering, 2020).
Most designs stop there, and the OTP is still late, because the lanes merge again at the provider:
- FCM's quota is per project. If bulk takes 590,000 of the 600,000 a minute, the OTP gets a 429.
- An SMS sender has one queue. Twilio queues what exceeds a sender's throughput (one segment a second for a US long code, 100+ for a short code), and "messages can only be queued for 10 hours and then they automatically fail" (Twilio docs). An OTP from the same number as a 500,000-text promotion joins the back.
So the lanes must reach the provider: reserved quota that bulk can't touch, separate SMS senders for OTPs and marketing, and a short validity period on time-boxed messages. Twilio's ValidityPeriod (1–36,000 seconds, default 36,000) drops a message that waits too long. A 5-minute OTP should carry no more than 300; Day 2 sets 120, so any code that arrives still has minutes left to be typed.
%% title: Priority lanes that stay separate up to the provider
flowchart TD
CR@{ icon: "sd:queue", label: "notifications.critical<br/>OTPs, security" }
TR@{ icon: "sd:queue", label: "notifications.transactional<br/>payments" }
LO@{ icon: "sd:queue", label: "notifications.bulk<br/>campaign, paced" }
WC["Critical workers<br/>own pool"]:::core
WT["Transactional workers"]:::service
WL["Bulk workers<br/>kill switch, ramp"]:::service
RB["Provider token buckets<br/>reserve for critical and transactional"]:::core
FCM@{ icon: "tech:firebase", label: "FCM<br/>600K a minute per project" }
OTP@{ icon: "sd:sms", label: "SMS sender A<br/>OTPs only, validity 120 s" }
MKT@{ icon: "sd:sms", label: "SMS sender B<br/>marketing" }
CR --> WC --> RB
TR --> WT --> RB
LO --> WL --> RB
RB --> FCM
RB --> OTP
RB -.->|"bulk only"| MKT
class CR,TR,LO queue
class FCM,OTP,MKT external# providers/quota.py: one bucket per provider, with a share bulk can't touch (illustrative)
import time
class ReservedBucket:
def __init__(self, rate_per_s, reserve_fraction=0.3):
self.rate, self.capacity = rate_per_s, rate_per_s # one second of burst
self.tokens, self.updated = rate_per_s, time.monotonic()
self.reserve = rate_per_s * reserve_fraction # kept for critical and transactional
def _refill(self):
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
self.updated = now
def try_take(self, priority) -> bool:
self._refill()
floor = self.reserve if priority == "bulk" else 0 # bulk stops at the reserve
if self.tokens - 1 >= floor:
self.tokens -= 1
return True
return False # caller waits or requeues
def on_429(self, retry_after_s=60):
"""Provider pushed back: halve our rate and spend nothing until Retry-After."""
self.rate = max(1, self.rate / 2)
self.tokens = -retry_after_s * self.rate # FCM: default 60 s if no headerSet the bucket below the provider's limit (FCM's 10,000 a second becomes perhaps 8,000), share it across workers through a central counter (Week 1, Day 3 · Rate Limiting), and halve it on a 429. FCM's rules: honour retry-after (default 60 seconds), never retry immediately ("wait at least 10 seconds"), and don't retry 400, 401, 403 or 404 at all. That's Week 3, Day 3's backpressure, with the provider as the slow consumer.
Chapter 4: When a Provider Fails#
4.1 How Each Provider Fails#
| Provider | What you see | What to do |
|---|---|---|
| APNs, outage | 500, 503 ServiceUnavailable or Shutdown, timeouts, GOAWAY |
Back off; Apple says 5xx sends can be retried after 15 minutes |
| APNs, our credentials | 403 ExpiredProviderToken |
Fix the signer: refresh "no more than once every 20 minutes and no less than once every 60 minutes" |
| APNs, bad tokens | 400 BadDeviceToken, 410 Unregistered |
Never retry; clean up (Chapter 7) |
| APNs, one device flooded | 429 TooManyRequests |
Retry with a delay, collapse |
| FCM | 429, 500, 503 | retry-after, backoff with jitter, at least 10 s |
| SMS, outage | 5xx, timeouts | Breaker; shift traffic to the other provider |
| SMS, silent stall | 200s keep coming, receipts slow | Breaker driven by receipt latency |
Apple warns that 4xx errors slow you down and "APNs disconnects a provider connection with too many error conditions", sooner for BadDeviceToken; a 410 isn't counted. Dead tokens hurt live ones.
4.2 Circuit Breakers per Provider#
The mechanics are in Week 2, Day 3 · Circuit Breakers. What's specific here:
- One breaker per provider and route: "provider A to India" can fail while "A to the US" works.
- Count the right failures: 5xx, timeouts and slow receipts. A 410 or an invalid number is our data, not a sick provider; counting them opens breakers during a token cleanup.
- Open means fail over or hold, never drop. Critical goes to the alternative; bulk waits or expires.
- Half-open lets a trickle through, not the backlog.
4.3 SMS Failover Without a Stampede#
SMS is the one channel with interchangeable providers. GOV.UK Notify's fix (2020):
- Always split. Each text picks a provider at random, about 50/50, so both stay warm and scaled.
- Shift in steps. A single 5xx cuts that provider's share by 10 points, at most once a minute. If "30% of them took longer than 4 minutes to report back as delivered" over the last 10 minutes, its share drops too.
- Drift back. If neither share changed for an hour, both move 10 points back towards the resting split.
# sms/balancer.py: GOV.UK Notify's published rules, as code (illustrative)
import random, time
class SmsBalancer:
def __init__(self, resting): # e.g. {"provider_a": 50, "provider_b": 50}
self.resting, self.share = dict(resting), dict(resting)
self.last_change = {p: 0.0 for p in resting}
def pick(self) -> str:
names = list(self.share)
return random.choices(names, weights=[self.share[p] for p in names])[0]
def shift_away(self, bad, points=10): # call on a 5xx, or a slow-receipt window
now = time.time()
if now - self.last_change[bad] < 60: # at most one cut a minute
return
moved = min(points, self.share[bad])
self.share[bad] -= moved
others = [p for p in self.share if p != bad]
for p in others:
self.share[p] += moved / len(others)
self.last_change[bad] = now
def hourly(self):
"""No change for an hour: move 10 points back towards the resting split."""
if time.time() - max(self.last_change.values()) >= 3600:
for p in self.share:
gap = self.resting[p] - self.share[p]
self.share[p] += max(-10, min(10, gap))The rule: move traffic in steps, never onto a provider that's had none.
4.4 When There's No Second Provider#
An iPhone takes notifications only from APNs; Android devices with Google Play services from FCM. When one is down, there's nowhere to fail over to.
APNs is down for 20 minutes
APNs returns 503 for 20 minutes from 20:00. Pending iOS pushes: 40,000 security and fraud alerts, 160,000 payment alerts, 1,000,000 campaign pushes. What do you do with each group, and what must you avoid when APNs comes back?
One strong answer
- Security and fraud alerts: SMS after a short wait, not the usual 2 unread minutes: the push can't go, and an alert minutes late is little use. ~40,000 texts, about $500 at ~$0.012 each (Chapter 5): cheap next to a fraud attempt nobody hears about. (OTPs aren't in the backlog: they go by SMS from the start.)
- Payments: in-app for connected users; hold the rest, collapsed per user so the backlog becomes one banner. They're in the inbox anyway.
- Campaign: pause; resume within its window or let it expire.
- On recovery: drain ramped, critical first. A million queued sends in one second is the Preface again, aimed at Apple.
The general rules: hold and retry with a per-notification expiry (Uber's push platform uses a "time to live value ranging from a few seconds to up to 30 minutes", 2020), use the socket for users with the app open, switch channel for critical only (SMS, on a budget: OTPs are on SMS already, and a security alert sends its SMS at once instead of after the usual 2 unread minutes), let bulk wait or expire, and ramp the drain on recovery.
4.5 Unknown Outcomes and Failover Duplicates#
A request that times out may have been sent, so a retry elsewhere can deliver twice.
What breaks if the OTP is generated per attempt?
The SMS worker builds the body, including a fresh OTP, inside each send attempt. Provider A times out after 8 seconds but actually delivered. The breaker opens; the retry goes to provider B, which delivers too. What does the user see, and what's the fix?
One strong answer
- Two texts, two codes. If the server kept only the second, the first one typed fails: a lockout caused by a retry.
- Fix: create the code once, before any send, and store it with the notification; every attempt on any provider sends the same body.
- And key the send ledger
(notification_id, channel), not by provider, so failover is a retry of the same send and a duplicate text is harmless.
The rule per class: OTPs retry only with the same code; payment alerts retry with the same collapse id; marketing isn't retried on timeout, since a missed promotion costs less than a duplicate.
Part III: Cost, Regions and Edge Cases#
Chapter 5: Cost Control#
5.1 SMS Is Most of the Bill#
Push and in-app cost compute; SMS is priced per segment. Twilio's list prices: $0.0083 per segment in the US plus carrier fees from $0.0035 (AT&T) to $0.005 (Verizon); $0.0832 per segment to India; $0.001 for each failed message (Twilio pricing, October 2026). Segment size depends on the alphabet (Twilio docs): GSM-7 fits 160 characters, 153 per part when split; UCS-2 fits 70, then 67. One character outside GSM-7, an emoji or a curly apostrophe, switches the whole message to UCS-2.
Count the segments
(a) A 95-character English OTP; (b) the same alert in Hindi, 95 characters; (c) a 150-character English payment alert whose template contains one curly apostrophe (’). How many segments each?
One strong answer
- (a) 1: GSM-7, under 160.
- (b) 2: Devanagari is UCS-2; 95 > 70, so it splits into 67-character parts.
- (c) 3: the ’ forces UCS-2; 150 ÷ 67 rounds up to 3. With a straight ' it's 1.
- So one character can triple the cost of every message from a template. Check segments before a template ships.
| Value | |
|---|---|
| SMS a day (Day 1: 5% of 500M) | 25M → 750M a month |
| Mix (assumed: a platform that hasn't yet applied Day 1's channel rules, so marketing still goes by SMS) | 10M / 10M / 5M a day |
| Segments: OTPs in English, 1 each | 10M |
| Payments: 70% English (1), 30% Hindi and other scripts (2) | 7M + 6M = 13M |
| Marketing: long messages, 2 each | 10M |
| Segments a day | 33M |
| Price per segment (Twilio US list $0.0083 + a $0.0035 carrier fee) | ~$0.012 |
| List-price spend | ~$396,000 a day, ~$11.9M a month |
| Lever 1: marketing moves to push, in-app and email digests | −10M segments |
| Lever 2: payment SMS only when push actually failed (assume half today are unnecessary) | −6.5M segments |
| After both | 16.5M segments → ~$198,000 a day, ~$5.9M a month |
Conclusion: the bill is messages × segments × price, and the platform controls the first two. Halving it needed no negotiation: SMS only for critical sends and real push failures, nothing marketing-shaped. Negotiated rates (Day 1 assumed them) change the totals, not the ratios; India's list price ($0.0832) is about 7× the ~$0.012 US price used here, so the levers matter more there. Every SMS should record why it was sent, so spend per hour can be split by type and alerted on.
5.2 Dedup and Digesting#
The cheapest notification is the one never sent:
- Dedup at every stage: Day 2's ingestion key, the campaign key, the send ledger. LinkedIn's ATC "handles the duplicated requests by dropping any request it has already processed" (2018).
- Digest what can wait. ATC can "send a group of notifications together as one notification at a later time"; with it, LinkedIn "cut member complaints in half" (2018).
- Collapse what replaces itself with
apns-collapse-idand FCM collapse keys. - Cap frequency per user across teams, as Uber's gateway does (2022).
5.3 The Spend Alert Is a Security Alert#
SMS pumping uses your phone-number fields, usually sign-up or OTP, to send texts to a range of numbers controlled by one mobile operator, and the attacker gets a share of the revenue those texts generate. Twilio's signs: "a spike of messages sent to a block of adjacent numbers", and no "completed verification cycle" (Twilio docs). So alert on spend per hour by country, track OTP conversion (codes entered ÷ codes sent) per country and prefix, rate-limit OTP sends per number, IP and device, and allow only countries you serve.
Chapter 6: Regional Failures#
6.1 What a Region Takes With It#
On 19–20 October 2025, a race condition left an empty DNS record for DynamoDB's endpoint in AWS us-east-1. Impact ran from 11:48 PM to 2:20 PM PDT; "between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed", and Lambda's SQS polling "failed and did not recover automatically" (AWS post-event summary, 2025). Two lessons: you can't scale up in a broken region, so failover capacity must already be running; and unconsumed queues become backlogs that hurt again on recovery.
21:00, region A goes dark
Region A, home to half the users, fails at 21:00. In flight: a payment alert marked sent in A at 20:59:58, a digest scheduled for 21:00 for users in Tokyo, and 5M open sockets. Replication lags 3 seconds. What happens to each when region B takes over, and what must B do before sending anything?
One strong answer
- Payment alert: its "sent" mark may not have replicated, so B sends it again: a duplicate, collapsed on the device. Better than losing it.
- Digest: the schedule replicated, so B can fire it, but only after fencing A (bumping a region epoch A's senders check). Otherwise a half-alive A sends it too.
- 5M sockets: they reconnect into B. Without jitter and admission control that's Day 3's storm at twice the size; with them, B admits everyone in a minute or two.
6.2 A Two-Region Design#
A typical design (ours, not any one company's):
- Each user has a home region; the outbox, inboxes and send ledger replicate to the other region asynchronously, seconds behind.
- On region loss, fence first: bump a region epoch so a half-alive region can't keep sending (Week 5, Day 5 · Leader Election).
- Replay unsent notifications, critical first, bulk last. "Sent" marks from the final seconds didn't replicate, so a few duplicates go out; collapse ids hide them on push.
- Re-admit sockets with Day 3's admission control; clients resume from the inbox.
- Provider callbacks must be able to land in either region.
%% title: Losing a region: fence, replay, re-admit
flowchart TD
subgraph R1["Region A · lost"]
A1["Pipeline for A's users"]:::bad
end
subgraph R2["Region B · survivor"]
F["Fence<br/>region epoch bumped"]:::core
RP["Replay unsent from the replicated outbox<br/>send ledger skips sent, critical first"]:::core
GW@{ icon: "sd:gateway", label: "Gateways<br/>admission control" }
DB@{ icon: "sd:database", label: "Replicated store<br/>outbox, inboxes, ledger" }
end
U@{ icon: "sd:users", label: "A's users<br/>reconnect with jitter" }
P@{ icon: "sd:cloud", label: "APNs, FCM, SMS<br/>global endpoints" }
A1 -.->|"async replication, seconds behind"| DB
F --> RP
DB --> RP
RP --> P
U --> GW
GW -->|"resume after last seq"| DB
class GW edge
class DB store
class U client
class P externalChapter 7: Edge Cases at Scale#
At 50M users, a one-in-a-million case happens 50 times a day.
7.1 Time Zones and Quiet Hours#
"No marketing between 22:00 and 08:00": whose 22:00?
- Store an IANA zone (
Asia/Kolkata), never a fixed UTC offset: a zone's offset changes with daylight saving. Update it from the device when the app opens, since travellers move. - Daylight saving makes missing and doubled times. In
America/New_York, 02:30 doesn't exist on the spring-forward day and 01:30 happens twice in autumn. A scheduled send must resolve to one real instant. - Rules follow the recipient. In the US, for example, telephone solicitations (marketing calls and messages, which includes texts) to residential subscribers are barred "before the hour of 8 a.m. or after 9 p.m. (local time at the called party's location)" (47 CFR 64.1200(c)(1)). The rule covers telemarketing, not push or transactional alerts, and messages sent with the recipient's permission or within an established business relationship fall outside its definition of a solicitation; but where a rule like it applies, a phone number's area code isn't where its owner is.
- Critical bypasses quiet hours: a 03:00 security alert is the point.
# routing/quiet_hours.py: next allowed send time, DST-safe (illustrative)
from datetime import datetime, time, timedelta
from zoneinfo import ZoneInfo
QUIET_START, QUIET_END = time(22, 0), time(8, 0)
def next_allowed(now_utc: datetime, tz_name: str, priority: str) -> datetime:
if priority == "critical":
return now_utc # security alerts ignore quiet hours
tz = ZoneInfo(tz_name)
local = now_utc.astimezone(tz)
if QUIET_END <= local.time() < QUIET_START:
return now_utc # outside quiet hours: send now
day = local.date() if local.time() < QUIET_END else local.date() + timedelta(days=1)
wake = datetime.combine(day, QUIET_END, tzinfo=tz) # 08:00 local, on the right date
return wake.astimezone(ZoneInfo("UTC")) # one real instant, even across DSTCampaign waves reuse it: release each zone's recipients when their window opens.
7.2 Uninstalls and Stale Tokens#
Nobody tells you about an uninstall; a send fails, sometimes weeks later.
- APNs 410
Unregisteredcarries atimestamp: when "APNs confirmed the token was no longer valid". Delete only if the token wasn't registered again after it. - FCM
UNREGISTERED(404) andINVALID_ARGUMENT(400) mean "it is safe to delete your record", butINVALID_ARGUMENT"signals an invalid registration only if the payload is completely valid" (Firebase docs). - Tokens also go stale silently. By default FCM treats a registration as stale "if its app instance hasn't connected for a month"; on Android, 270 days inactive means expired. Firebase recommends refreshing token timestamps about monthly and pruning stale ones, which otherwise appear as "a dramatic (but not actually valid) drop in delivery rates".
# push/tokens.py: act on what providers say about tokens (illustrative)
def on_push_result(token, r):
if r.provider == "apns" and r.status == 410:
if token.registered_at_ms <= r.timestamp_ms: # dead since our last registration
tokens.mark_dead(token.id, r.reason)
elif r.provider == "apns" and r.reason == "BadDeviceToken":
tokens.mark_dead(token.id, "bad_token") # often a sandbox token in production
elif r.provider == "fcm" and r.error == "UNREGISTERED":
tokens.mark_dead(token.id, "unregistered")
elif r.provider == "fcm" and r.error == "INVALID_ARGUMENT" and payload_is_valid(r.notification):
tokens.mark_dead(token.id, "invalid") # only then is the token to blameAn uninstall is a choice: fall back to SMS for critical notifications only. And watch invalidation by app version: a doubling on release day means the release broke registration.
7.3 Duplicate Sends Across Retries#
Queues redeliver, workers time out, breakers fail over, regions replay. Exactly-once delivery to a phone isn't available; at-least-once with dedup and collapse is (Week 2, Day 2 · Idempotency in Practice):
| Duplicate source | What stops it |
|---|---|
| Producer retries the API | Ingestion idempotency key (Day 2) |
| Campaign job restarts | (campaign_id, user_id, channel) |
| Redelivery after a worker crash | Send ledger (notification_id, channel): record "attempting", send, record "sent" |
| Timeout, then retry or failover | Same body and code every attempt; collapse ids |
| Region replay | Ledger, plus collapse ids for what slips through |
Graph dedup hits per stage: a jump means a worker is crash-looping or a breaker is flapping.
Part IV: Real-World Application#
Chapter 8: How Real Companies Run It#
- GOV.UK Notify (2020): an always-split SMS pair, shifted in steps; "between 100,000 and 200,000 text messages a day".
- LinkedIn ATC (2018): one gateway for email, push and SMS with rate limits, digests, locale-aware timing and dedup; "over a billion requests per day".
- Uber (2020, 2022): a push platform at 1.5M concurrent connections and 250,000 messages a second with three priorities, and a gateway enforcing per-user caps.
- Netflix RENO (2022): priority-specific queues and clusters, and a staleness filter.
Chapter 9: Common Mistakes#
- Measuring the worker, not the user. "Send took 180 ms" while the message waited 4 minutes. Measure from
created_at; watch lane age. - Lanes that end at your border. Separate topics, then one FCM quota and one SMS sender: the OTP waits at the provider. Reserve quota, split senders.
- All-or-nothing failover. 100% of traffic onto a cold provider repeats GOV.UK Notify's 2019 failure. Stay split; shift in steps.
- Counting bad tokens as provider failures. A token cleanup opens the breaker on a healthy APNs.
- No owner for SMS spend. A retry loop or pumping attack shows up on next month's invoice.
Part V: Interview Preparation#
Chapter 10: The Whole Week in One Interview#
10.1 The 45 Minutes#
"Design a notification platform" is a common prompt; this week is the full answer.
| Minutes | What you do | Week 6 |
|---|---|---|
| 0–5 | Requirements: types, channels, 50M users, 500M a day, 10M an hour in campaigns, latency per class, 99.9% for critical, marketing never delays transactions | Day 1 |
| 5–10 | Estimates: ~5,800 a second average, ~17,000 peak, channel mix, SMS cost, 10M sockets at the evening peak | Days 1, 3, 5 |
| 10–20 | Architecture: API with idempotency keys → outbox → router → lanes → channel workers → providers; inbox and gateways for in-app | Days 1, 2 |
| 20–35 | Deep dive, interviewer's pick: delivery guarantees, real-time delivery and reconnect storms, presence and group fan-out | Days 2, 3, 4 |
| 35–45 | Operations: SLOs per class, lanes to the provider, breakers and SMS failover, cost, regions, edge cases | Day 5 |
%% title: The week's platform, end to end
flowchart TD
SVC@{ icon: "sd:server", label: "Product services<br/>payments, security, marketing" }
API["Notification API<br/>idempotency key"]:::core
OB@{ icon: "sd:database", label: "Notification store and outbox" }
RT["Router<br/>preferences, quiet hours, caps, dedup"]:::core
LN@{ icon: "sd:queue", label: "Lanes<br/>critical · transactional · bulk" }
CW["Channel workers<br/>token buckets, breakers, send ledger"]:::service
APNS@{ icon: "tech:apple", label: "APNs" }
FCM@{ icon: "tech:firebase", label: "FCM" }
SMS@{ icon: "sd:sms", label: "SMS providers A and B<br/>weighted split" }
EM@{ icon: "tech:sendgrid", label: "Email provider" }
IB@{ icon: "sd:database", label: "In-app inbox<br/>seq per user" }
GW@{ icon: "sd:gateway", label: "Gateways<br/>WebSocket and SSE" }
CH["Chat owners, presence shards<br/>Day 4"]:::service
DEV@{ icon: "sd:phone", label: "Apps and browsers" }
OBS@{ icon: "sd:dashboard", label: "SLOs, lane age, spend<br/>alerts and runbooks" }
SVC --> API --> OB --> RT --> LN --> CW
CW --> APNS
CW --> FCM
CW --> SMS
CW --> EM
CW --> IB --> GW
CH --> GW
GW <--> DEV
APNS --> DEV
FCM --> DEV
CW -.-> OBS
class SVC,DEV client
class OB,IB store
class LN queue
class APNS,FCM,SMS,EM external
class GW edge
class OBS service10.2 Key Phrases#
MEASURING
"End-to-end from creation time, and the oldest message's age per lane. Success rates can't see a queue."
PROMISING
"SLOs stop at the provider, because APNs doesn't report the device: 99.9% of critical handed off within 10 seconds. That's about 14 minutes of stall at peak a month, so critical gets its own workers and quota."
BURSTS AND PROVIDERS
"Campaigns are paced, pausable jobs in their own lane, and lanes continue to the provider: reserved quota, OTPs on their own SMS sender. SMS is always split across two providers and shifted in steps, never failed over all at once."
10.3 Common Follow-Up Questions#
| Question | Strong answer |
|---|---|
| "How do you know a push was delivered?" | "Mostly you don't: we measure hand-off, SMS receipts, in-app acks and opens." |
| "What protects OTPs from a 10M campaign?" | "Separate lanes, pools and reserved provider quota; a paced, pausable campaign." |
| "APNs is down." | "Hold with TTLs, in-app if connected, SMS for critical only, ramp the drain." |
| "How do you avoid sending twice?" | "Dedup keys, a send ledger, the same body every attempt, collapse ids." |
Ask the interviewer: which notifications are critical, and how late is "late"? One region or several? Is there an SMS budget?
Chapter 11: Practice Problems#
Practice · problem 1
Problem 1: Election-Night Alerts#
Setup: Push "results declared" to 30M users within two minutes, while breaking-news alerts to 200,000 journalists stay under 10 seconds. What do you do the day before, and during?
Click for solution outline
- 250,000 a second, 25× FCM's default quota: ask Firebase well ahead, precompute the audience, pre-warm workers and APNs connections.
- Lanes: journalists critical with reserved quota; the mass alert ramped in its own lane, one collapse id per race.
Practice · problem 2
Problem 2: The Silent SMS Provider#
Setup: Provider A returns 200s, but 40% of its receipts to Indian numbers take over 5 minutes. The breaker never opens. Why, and what changes?
Click for solution outline
- The breaker only counts HTTP status. Add receipt latency as a failure signal, GOV.UK Notify style, scoped per route: "A to India" loses share, "A to the US" keeps it.
- OTPs get a short validity period, so stuck texts die instead of arriving after the code expires.
Practice · problem 3
Problem 3: The Release That Lost iOS#
Setup: After an iOS release, token invalidation for that version is 30× normal, mostly BadDeviceToken. Diagnose, contain, prevent.
Click for solution outline
- Diagnosis: the build registers sandbox tokens, which production APNs rejects.
- Contain: route those users' critical notifications in-app or by SMS, ship a fix, and stop sending to the bad tokens: many
BadDeviceTokenerrors get connections dropped sooner. - Prevent: alert on invalidation by app version, and make canaries watch it.
Summary#
MEASURE AND PROMISE
- End-to-end from
created_atand lane age; SLOs stop at the provider hand-off
BURSTS AND PROVIDERS
- Paced, pausable campaigns; lanes reach the provider via reserved quota and separate SMS senders
- Breakers per provider and route; SMS always split, shifted in steps
COST, REGIONS, EDGE CASES
- SMS = messages × segments × price; cut the first two
- Regions: fence, replay from the ledger, re-admit slowly
- IANA zones, careful token deletion, same body on every retry
Further Reading#
Engineering Posts#
- Blog
GOV.UK Notify — How GOV.UK Notify reliably sends text messages to users (April 2020): https://gds.blog.gov.uk/2020/04/03/how-gov-uk-notify-reliably-sends-text-messages-to-users
- Blog
LinkedIn — Air Traffic Controller: Member-First Notifications at LinkedIn (March 2018): https://www.linkedin.com/blog/engineering/messaging-notifications/air-traffic-controller-member-first-notifications-at-linkedin
- Blog
Uber — How Uber Optimizes Push Notifications Using ML (November 2022): https://www.uber.com/blog/how-uber-optimizes-push-notifications-using-ml/
- Blog
Uber — Uber's Real-Time Push Platform (December 2020): https://www.uber.com/blog/real-time-push-platform/
- Docs
Netflix — Rapid Event Notification System at Netflix (2022), via InfoQ: https://www.infoq.com/news/2022/03/netflix-reno
- Blog
Duolingo — Super Bowl commercial 2024 (February 2024): https://blog.duolingo.com/super-bowl-commercial-2024
- Read
AWS — DynamoDB service disruption in us-east-1 (October 2025): https://aws.amazon.com/message/101925/
Provider Documentation#
- Docs
Apple — Sending notification requests to APNs, Handling notification responses from APNs, Establishing a token-based connection to APNs: https://developer.apple.com/documentation/usernotifications/sending-notification-requests-to-apns · https://developer.apple.com/documentation/usernotifications/handling-notification-responses-from-apns · https://developer.apple.com/documentation/usernotifications/establishing-a-token-based-connection-to-apns
- Docs
Firebase — Throttling and quotas, Sending at scale, Token management, Understanding delivery: https://firebase.google.com/docs/cloud-messaging/throttling-and-quotas · https://firebase.google.com/docs/cloud-messaging/scale-fcm · https://firebase.google.com/docs/cloud-messaging/manage-tokens · https://firebase.google.com/docs/cloud-messaging/understand-delivery
- Docs
Twilio — Segments, queue overflow (30001), Message resource, SMS pumping, pricing: https://www.twilio.com/docs/glossary/what-sms-character-limit · https://www.twilio.com/docs/api/errors/30001 · https://www.twilio.com/docs/messaging/api/message-resource · https://www.twilio.com/docs/glossary/what-is-sms-pumping-fraud · https://www.twilio.com/en-us/sms/pricing/us · https://www.twilio.com/en-us/sms/pricing/in
- Read
47 CFR 64.1200 (US telephone solicitation rules): https://www.law.cornell.edu/cfr/text/47/64.1200
this lesson in shorts · 4
play them all →One idea from this lesson in about a minute each, made for your phone.
Go deeper · reading trail
all reading →Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.
How GOV.UK Notify reliably sends text messages to users
David McDonald · 2020- What
- How the UK government's notification service sends 100,000 to 200,000 texts a day through two SMS providers.
- Key idea
- Failing all traffic over to a cold provider broke it too, so every text now picks a provider at random, about 50/50, and errors or slow receipts shift 10 points at a time.
- Why here
- The Preface and SMS failover chapter of this lesson: stay split, shift in steps, and never move a whole spike onto a provider that hasn't seen it.
T For Tsunami : Dealing with traffic spikes
Akash Saxena · 2017- What
- Hotstar's retrospective on traffic spikes during the 2017 Champions Trophy.
- Key idea
- A push notification sent as Virat Kohli walked in to bat took login traffic from 10K to 125K TPS in seconds, and the database had not been scaled for it.
- Why here
- Why this lesson treats a campaign as a paced, planned job: a mass notification is also a load test on everything users land on.
Kept from the earlier trail (checked 2026-10-18); blog.hotstar.com answered the 2026-10-29 automated re-check with 403.
Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region
2025- What
- AWS's post-event summary of the October 2025 DynamoDB disruption in us-east-1 and its knock-on effects.
- Key idea
- A DNS race condition left DynamoDB's regional endpoint with an empty record; new EC2 launches then failed for hours and Lambda's SQS polling had to be restored by hand.
- Why here
- This lesson's regional-failure chapter: you can't scale up in a broken region, and queues left unconsumed become backlogs that hurt again on recovery.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.



