~learn.
?

Day 0528 minPreface+60 XP

Operations, Scale and Edge Cases.

5 stages5 code

watch it first · newThe whole lesson in a 24-minute animated lecture5 quiz checkpoints · ask the AI about any moment · captions, chapters and speed

Preface · 2 min

Preface#

GOV.UK Notify sends text messages for UK central government, local authorities and the NHS: flood warnings, sign-in codes. It used two SMS providers. All traffic went through one; if it returned a single 5xx or its delivery receipts slowed, Notify swapped all traffic to the other automatically. The team rebalanced by hand "maybe once a week" (GOV.UK Notify, GDS blog, April 2020). Then, towards the end of 2019:

GOV.UK Notify, late 2019 (from the team's April 2020 post)
  • A spike arrives
    • "A large spike in requests to send text messages" goes to one provider, which "couldn't handle the load and started to fail"
  • Failover works as designed
    • Notify swaps all traffic to the second provider
  • The second provider fails too
    • "Sending a large amount of traffic out of nowhere caused them to start returning errors too"; the providers likely "needed time to scale up"
  • The fix
    • Every text now goes to a provider picked at random, roughly 50/50, and errors shift traffic 10 percentage points at a time

By the team's own account, the providers most likely just needed time to scale up: the platform's own traffic shape broke both. The failover was correct and still made things worse, because it moved the whole spike at once onto a provider that hadn't seen it coming.

Day 1 designed the platform, Day 2 its delivery pipeline, Day 3 the gateways and Day 4 presence and fan-out. Today we run it: what to measure and promise, how to survive campaigns, provider outages and lost regions, what it costs, and the edge cases of millions of users. Then we walk the whole week as one interview answer.

ExplainerThe whole lesson: operations, scale and edge cases

Day 5 runs the platform. GOV.UK Notify's all-or-nothing SMS failover broke both providers, so traffic is split and shifted in steps. Measure the delivery funnel (accepted is not delivered), latency from created_at and lane age; SLOs stop at the provider hand-off, and ~14 minutes of stall at peak spends a month's critical budget. Campaigns are paced jobs (~5,500/s from 10:07), lanes reach the provider through reserved quota and a separate OTP sender, breakers work per provider and route, and an APNs outage sends only critical alerts to SMS. SMS cost is messages × segments × price (~$11.9M to ~$5.9M a month); a lost region is fenced, replayed and re-admitted; and the week closes as one 45-minute interview.

Part I · Stage 1 of 5 · 5 min

Part I: Knowing What's Happening#

Chapter 1: What to Measure#

1.1 The Simple Definition#

Operating the platform means answering three questions at any moment, with numbers: are notifications arriving on time (per class: an OTP and a sale banner have different clocks), where are they stuck (our queues, a provider, the device), and what is it costing (SMS money, user patience, provider goodwill)?

1.2 The Delivery Funnel#

Count every notification at every stage, labelled by priority class, channel and provider, and the funnel shows where it's stuck.

DiagramOne notification's funnel, and where each metric is measured
%% title: One notification's funnel, and where each metric is measured
flowchart TD
  A["Accepted by the API<br/>created_at stamped"]:::core --> B["Routed<br/>preferences, quiet hours, dedup"]:::service
  B --> C@{ icon: "sd:queue", label: "Lane topic<br/>oldest message age" }
  C --> D["Channel worker<br/>rate limiter, breaker"]:::service
  D --> E@{ icon: "tech:apple", label: "APNs<br/>200 or an error" }
  D --> F@{ icon: "tech:firebase", label: "FCM<br/>200 or an error" }
  D --> G@{ icon: "sd:sms", label: "SMS provider<br/>queued, then a receipt" }
  D --> H@{ icon: "sd:gateway", label: "Gateways<br/>client ack by seq" }
  E -.->|"no receipt"| I@{ icon: "sd:phone", label: "Device<br/>shown, opened" }
  F -.->|"Android data, days late"| I
  G -.->|"delivery receipt"| I
  H -->|"ack"| I
  class C queue
  class E,F,G external
  class H edge
  class I client

The dotted edges are the hard part: "provider accepted" isn't "delivered", and the providers say so.

  • APNs is "a best-effort service"; notifications "may also get throttled, saved in storage, and in some cases, not delivered" (Apple docs).
  • FCM counts as sent anything "enqueued for delivery or ... successfully passed to a third-party service"; its delivery data covers Android only and is "delayed by up to 5 days" (Firebase docs).
  • SMS has receipts: Twilio's sent means "the nearest upstream carrier accepted the outbound message", delivered that it "received confirmation of outbound message delivery" (Twilio docs).
  • In-app is the one channel where we see the device: the client acks the sequence number (Day 3).

1.3 The Signals Worth a Graph#

Signal How to measure it Why
End-to-end delivery latency Provider accepted (or client ack) minus created_at, p50/p99 per class and channel The user's clock
Oldest message age per lane Oldest unconsumed message in each priority topic 2M in the bulk lane can be fine; 2,000 in critical isn't
Send success by channel and provider Accepted ÷ attempted, split retryable (429, 5xx, timeouts) and permanent (bad token, bad number) Retryable: the provider is unwell. Permanent: our data is
SMS receipt rate and latency delivered ÷ sent, time to receipt, per provider and country 200s can hide stalled carriers
Fan-out lag (Day 4) Log append to last gateway publish; inbox pointer lag Hot rooms show here first
Inbox sync backlog (Days 3–4) Devices with cursors far behind; sync duration Rises after reconnect waves
Connections (Day 3) Sockets per gateway, reconnects per second, admission refusals, ack-timeout fallbacks Storms show here first
Presence pushes per change (Day 4) Presence events delivered ÷ state changes Leaky subscriptions
Push token invalidation rate APNs 410 and BadDeviceToken, FCM UNREGISTERED per 1,000 sends, by app version A jump after a release is a bug
Spend per channel per hour Segments × price, by type and country Money, and fraud

Two rules keep these honest. Measure from created_at, not dequeue time: a worker timing "picked up to sent" reports 200 ms while messages wait 4 minutes. And keep labels small: ids go in logs and traces (Week 10, Day 2 · Observability).

pythonrecord · 14 lines
# workers/metrics.py: record the user's clock, not the worker's (illustrative)
import time
from prometheus_client import Histogram, Counter

E2E = Histogram("notif_e2e_seconds", "created_at to provider accepted",
                ["priority", "channel", "provider"],
                buckets=(0.5, 1, 2, 5, 10, 30, 60, 300, 900, 3600))
SENDS = Counter("notif_sends_total", "send attempts by outcome",
                ["priority", "channel", "provider", "outcome"])   # ok | retryable | permanent

def record(n, provider, outcome):
    SENDS.labels(n.priority, n.channel, provider, outcome).inc()
    if outcome == "ok":
        E2E.labels(n.priority, n.channel, provider).observe(time.time() - n.created_at)
Your turn · checkpoint 1not answered

Green dashboard, angry users

It's 19:05. Push success is 99.8% and errors are flat, but support is flooded: OTPs take four minutes. Which metric would have shown it, and why did the success rate stay green?

Think it through, or jot it down. Then compare.

One strong answer

  • Oldest message age in the critical lane, or p99 end-to-end latency for critical: either jumps within a minute.
  • Success rates only count attempts. A message waiting in a topic hasn't failed, so a four-minute queue looks perfect until it times out.
  • The usual cause: a campaign sharing the OTPs' lane, workers or provider quota (Chapter 3).

Chapter 2: SLOs for Delivery#

2.1 What to Promise#

Week 10, Day 1 · SLIs, SLOs and SLAs teaches the method. Here, an SLO must cover what we control and can measure; we can't see an iPhone show a banner, so push SLIs stop at the provider's acceptance. Day 1 set the targets: transactions in 1–2 seconds, security alerts near-instant, 99.9% for critical.

Class SLI SLO (30 days)
Critical (OTPs, security) Handed to a provider, or acked in-app, within 10 s of created_at 99.9%
Transactional (payments) Handed to a provider within 30 s 99.5%
In-app, client connected Client ack within 2 s of the inbox write 99%
SMS receipts, critical delivered within 2 minutes of sent, per provider 98%
Bulk (campaigns, digests) Sent within the campaign window, never in quiet hours 99% of recipients

Bulk has no availability target: pausing a campaign to protect OTPs is the system working. The SMS receipt SLO is partly outside our control, so it drives failover (Chapter 4).

Your turn · checkpoint 2not answered

How long can the critical lane stall?

Assume critical notifications (OTPs and security alerts) are 5% of Day 1's 500M a day; Day 1 doesn't split volume by class, so this share is illustrative. The SLO is 99.9% within 10 seconds over 30 days, and peak is 3× average. How many late critical notifications does a month allow, and how long a full stall at peak spends all of it?

Think it through, or jot it down. Then compare.

One strong answer

  • Volume: 25M a day × 30 = 750M. Budget: 0.1% = 750,000 late notifications.
  • Peak: 25M ÷ 86,400 ≈ 290 a second, × 3 ≈ 870 a second.
  • Stall: 750,000 ÷ 870 ≈ 860 s: about 14 minutes at peak spends the month.
  • So one bad deploy or one shared provider queue can burn a month in a lunch break: the critical lane gets its own workers and quota, and a fast-burn page.

2.2 What Pages#

Page on symptoms users feel, with Week 10's multi-window burn-rate alerts (14.4× burn over an hour pages at once), plus the few causes that need a human within minutes:

Alert Severity Why
Critical-lane SLO fast burn Page Users can't sign in
Breaker open on a critical-path provider Page Failover is running
Any APNs ExpiredProviderToken (403) Page Apple rejects provider tokens over an hour old: our signer died, every iOS push fails
Connections down 5% in a minute Page Day 3's reconnect storm
SMS spend 3× the same hour last week Page A retry loop or SMS pumping
Token invalidation 2× after a release Ticket A registration bug
Bulk lane past its window Ticket Marketing is late, nothing is broken
Runbook: critical-lane SLO burning
  1. Lane age high? Our side: pause any campaign, then check critical workers and deploys.
  2. Provider slow or failing? 429 → slow down; 5xx or timeouts → check the breaker and failover.
  3. APNs 403s? Check the provider-token signing job and key first.
  4. Tell support what users see; update every 15 minutes.

Part II · Stage 2 of 5 · 8 min

Part II: Bursts and Provider Failures#

Chapter 3: Campaign Mode and Priority Lanes#

3.1 The Shape of a Campaign#

Day 1 allows 10M marketing notifications an hour on top of ~5,800 a second: only 2,800 a second more if spread evenly, but someone clicks Send and 10M rows land at once.

Bursts can be planned. During the 2024 Super Bowl, Duolingo pushed to 4 million learners, 95% within 3.9 seconds, after engineers had to "redesign the entire system to support the volume" (Duolingo, February 2024). A planned burst is an event; an unplanned one is an incident.

FCM says how it wants to be treated (Firebase docs):

  • Default quota: 600,000 messages a minute per project (10,000 a second); above it, 429 RESOURCE_EXHAUSTED.
  • Ramp: "at minimum, ramp from 0 to the max RPS across a 60 second time-window."
  • Avoid the quarter hours: FCM gets "more than double traffic during the first 30 seconds to 2 minutes of each hour", so stay out of "a 2 minute window of each of the :00, :15, :30, and :45 minute marks."
  • Per Android device: 240 messages a minute and 5,000 an hour, and "don't routinely send messages near this maximum rate."

3.2 Campaign Mode#

A campaign is a job, not 10M API calls:

  1. Snapshot the audience from a replica or warehouse, never the primary.
  2. Filter once: consent, preferences, frequency caps, quiet hours. Uber's Consumer Communication Gateway schedules each user's pushes under limits like "at most 2 pushes per day" and "8 hours between pushes" (Uber Engineering, 2022); LinkedIn's Air Traffic Controller "will rate-limit upstream applications to prevent them from accidentally spamming members" (LinkedIn Engineering, 2018).
  3. Pace into the bulk lane: 10M over 30 minutes ≈ 5,500 a second, ramped, starting at 10:07, not 10:00.
  4. Wave by time zone (Chapter 7).
  5. Key each send (campaign_id, user_id, channel) so a restarted job doesn't send twice.
  6. Kill switch: pause, resume, cancel, checked before every batch.

3.3 Priority Lanes All the Way to the Provider#

Day 1's interviewer: "Transaction and security notifications should never be delayed by marketing campaigns." So: a topic per class (notifications.critical, notifications.transactional, notifications.bulk), a worker pool per lane scaled separately, and lag watched per lane.

ExplainerPriority lanes, all the way to the provider

In one shared queue, a 10M campaign that lands at once puts 9.2M messages ahead of Rohan's 20:00:40 OTP, which leaves at ~20:08 after its code has expired. With a topic and worker pool per class (notifications.critical, notifications.transactional, notifications.bulk), the campaign is paced into bulk at ~5,500 a second and nothing is ever in front of the OTP. The lanes still merge at the provider (FCM's 600,000 a minute per project, one queue per SMS sender), so token buckets reserve a share bulk can't touch (Day 2 assumes 4,000/s, leaving 6,000/s for bulk) and OTPs get their own SMS sender with a 120 s validity period.

Netflix's RENO "segments incoming events by priority and routes them to priority-specific AWS SQS queues and the corresponding compute instance clusters", so one can fail without the others (Netflix Technology Blog, 2022, via InfoQ). Uber's push platform writes to the socket "in descending order of priority" (Uber Engineering, 2020).

Most designs stop there, and the OTP is still late, because the lanes merge again at the provider:

  • FCM's quota is per project. If bulk takes 590,000 of the 600,000 a minute, the OTP gets a 429.
  • An SMS sender has one queue. Twilio queues what exceeds a sender's throughput (one segment a second for a US long code, 100+ for a short code), and "messages can only be queued for 10 hours and then they automatically fail" (Twilio docs). An OTP from the same number as a 500,000-text promotion joins the back.

So the lanes must reach the provider: reserved quota that bulk can't touch, separate SMS senders for OTPs and marketing, and a short validity period on time-boxed messages. Twilio's ValidityPeriod (1–36,000 seconds, default 36,000) drops a message that waits too long. A 5-minute OTP should carry no more than 300; Day 2 sets 120, so any code that arrives still has minutes left to be typed.

DiagramPriority lanes that stay separate up to the provider
%% title: Priority lanes that stay separate up to the provider
flowchart TD
  CR@{ icon: "sd:queue", label: "notifications.critical<br/>OTPs, security" }
  TR@{ icon: "sd:queue", label: "notifications.transactional<br/>payments" }
  LO@{ icon: "sd:queue", label: "notifications.bulk<br/>campaign, paced" }
  WC["Critical workers<br/>own pool"]:::core
  WT["Transactional workers"]:::service
  WL["Bulk workers<br/>kill switch, ramp"]:::service
  RB["Provider token buckets<br/>reserve for critical and transactional"]:::core
  FCM@{ icon: "tech:firebase", label: "FCM<br/>600K a minute per project" }
  OTP@{ icon: "sd:sms", label: "SMS sender A<br/>OTPs only, validity 120 s" }
  MKT@{ icon: "sd:sms", label: "SMS sender B<br/>marketing" }
  CR --> WC --> RB
  TR --> WT --> RB
  LO --> WL --> RB
  RB --> FCM
  RB --> OTP
  RB -.->|"bulk only"| MKT
  class CR,TR,LO queue
  class FCM,OTP,MKT external
pythonReservedBucket · 26 lines
# providers/quota.py: one bucket per provider, with a share bulk can't touch (illustrative)
import time

class ReservedBucket:
    def __init__(self, rate_per_s, reserve_fraction=0.3):
        self.rate, self.capacity = rate_per_s, rate_per_s       # one second of burst
        self.tokens, self.updated = rate_per_s, time.monotonic()
        self.reserve = rate_per_s * reserve_fraction            # kept for critical and transactional

    def _refill(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
        self.updated = now

    def try_take(self, priority) -> bool:
        self._refill()
        floor = self.reserve if priority == "bulk" else 0       # bulk stops at the reserve
        if self.tokens - 1 >= floor:
            self.tokens -= 1
            return True
        return False                                            # caller waits or requeues

    def on_429(self, retry_after_s=60):
        """Provider pushed back: halve our rate and spend nothing until Retry-After."""
        self.rate = max(1, self.rate / 2)
        self.tokens = -retry_after_s * self.rate                # FCM: default 60 s if no header

Set the bucket below the provider's limit (FCM's 10,000 a second becomes perhaps 8,000), share it across workers through a central counter (Week 1, Day 3 · Rate Limiting), and halve it on a 429. FCM's rules: honour retry-after (default 60 seconds), never retry immediately ("wait at least 10 seconds"), and don't retry 400, 401, 403 or 404 at all. That's Week 3, Day 3's backpressure, with the provider as the slow consumer.

Chapter 4: When a Provider Fails#

4.1 How Each Provider Fails#

Provider What you see What to do
APNs, outage 500, 503 ServiceUnavailable or Shutdown, timeouts, GOAWAY Back off; Apple says 5xx sends can be retried after 15 minutes
APNs, our credentials 403 ExpiredProviderToken Fix the signer: refresh "no more than once every 20 minutes and no less than once every 60 minutes"
APNs, bad tokens 400 BadDeviceToken, 410 Unregistered Never retry; clean up (Chapter 7)
APNs, one device flooded 429 TooManyRequests Retry with a delay, collapse
FCM 429, 500, 503 retry-after, backoff with jitter, at least 10 s
SMS, outage 5xx, timeouts Breaker; shift traffic to the other provider
SMS, silent stall 200s keep coming, receipts slow Breaker driven by receipt latency

Apple warns that 4xx errors slow you down and "APNs disconnects a provider connection with too many error conditions", sooner for BadDeviceToken; a 410 isn't counted. Dead tokens hurt live ones.

4.2 Circuit Breakers per Provider#

The mechanics are in Week 2, Day 3 · Circuit Breakers. What's specific here:

  • One breaker per provider and route: "provider A to India" can fail while "A to the US" works.
  • Count the right failures: 5xx, timeouts and slow receipts. A 410 or an invalid number is our data, not a sick provider; counting them opens breakers during a token cleanup.
  • Open means fail over or hold, never drop. Critical goes to the alternative; bulk waits or expires.
  • Half-open lets a trickle through, not the backlog.

4.3 SMS Failover Without a Stampede#

SMS is the one channel with interchangeable providers. GOV.UK Notify's fix (2020):

  • Always split. Each text picks a provider at random, about 50/50, so both stay warm and scaled.
  • Shift in steps. A single 5xx cuts that provider's share by 10 points, at most once a minute. If "30% of them took longer than 4 minutes to report back as delivered" over the last 10 minutes, its share drops too.
  • Drift back. If neither share changed for an hour, both move 10 points back towards the resting split.
pythonSmsBalancer · 29 lines
# sms/balancer.py: GOV.UK Notify's published rules, as code (illustrative)
import random, time

class SmsBalancer:
    def __init__(self, resting):                 # e.g. {"provider_a": 50, "provider_b": 50}
        self.resting, self.share = dict(resting), dict(resting)
        self.last_change = {p: 0.0 for p in resting}

    def pick(self) -> str:
        names = list(self.share)
        return random.choices(names, weights=[self.share[p] for p in names])[0]

    def shift_away(self, bad, points=10):        # call on a 5xx, or a slow-receipt window
        now = time.time()
        if now - self.last_change[bad] < 60:      # at most one cut a minute
            return
        moved = min(points, self.share[bad])
        self.share[bad] -= moved
        others = [p for p in self.share if p != bad]
        for p in others:
            self.share[p] += moved / len(others)
        self.last_change[bad] = now

    def hourly(self):
        """No change for an hour: move 10 points back towards the resting split."""
        if time.time() - max(self.last_change.values()) >= 3600:
            for p in self.share:
                gap = self.resting[p] - self.share[p]
                self.share[p] += max(-10, min(10, gap))

The rule: move traffic in steps, never onto a provider that's had none.

4.4 When There's No Second Provider#

An iPhone takes notifications only from APNs; Android devices with Google Play services from FCM. When one is down, there's nowhere to fail over to.

Your turn · checkpoint 3not answered

APNs is down for 20 minutes

APNs returns 503 for 20 minutes from 20:00. Pending iOS pushes: 40,000 security and fraud alerts, 160,000 payment alerts, 1,000,000 campaign pushes. What do you do with each group, and what must you avoid when APNs comes back?

Think it through, or jot it down. Then compare.

One strong answer

  • Security and fraud alerts: SMS after a short wait, not the usual 2 unread minutes: the push can't go, and an alert minutes late is little use. ~40,000 texts, about $500 at ~$0.012 each (Chapter 5): cheap next to a fraud attempt nobody hears about. (OTPs aren't in the backlog: they go by SMS from the start.)
  • Payments: in-app for connected users; hold the rest, collapsed per user so the backlog becomes one banner. They're in the inbox anyway.
  • Campaign: pause; resume within its window or let it expire.
  • On recovery: drain ramped, critical first. A million queued sends in one second is the Preface again, aimed at Apple.

The general rules: hold and retry with a per-notification expiry (Uber's push platform uses a "time to live value ranging from a few seconds to up to 30 minutes", 2020), use the socket for users with the app open, switch channel for critical only (SMS, on a budget: OTPs are on SMS already, and a security alert sends its SMS at once instead of after the usual 2 unread minutes), let bulk wait or expire, and ramp the drain on recovery.

4.5 Unknown Outcomes and Failover Duplicates#

A request that times out may have been sent, so a retry elsewhere can deliver twice.

Your turn · checkpoint 4not answered

What breaks if the OTP is generated per attempt?

The SMS worker builds the body, including a fresh OTP, inside each send attempt. Provider A times out after 8 seconds but actually delivered. The breaker opens; the retry goes to provider B, which delivers too. What does the user see, and what's the fix?

Think it through, or jot it down. Then compare.

One strong answer

  • Two texts, two codes. If the server kept only the second, the first one typed fails: a lockout caused by a retry.
  • Fix: create the code once, before any send, and store it with the notification; every attempt on any provider sends the same body.
  • And key the send ledger (notification_id, channel), not by provider, so failover is a retry of the same send and a duplicate text is harmless.

The rule per class: OTPs retry only with the same code; payment alerts retry with the same collapse id; marketing isn't retried on timeout, since a missed promotion costs less than a duplicate.

Part III · Stage 3 of 5 · 7 min

Part III: Cost, Regions and Edge Cases#

Chapter 5: Cost Control#

5.1 SMS Is Most of the Bill#

Push and in-app cost compute; SMS is priced per segment. Twilio's list prices: $0.0083 per segment in the US plus carrier fees from $0.0035 (AT&T) to $0.005 (Verizon); $0.0832 per segment to India; $0.001 for each failed message (Twilio pricing, October 2026). Segment size depends on the alphabet (Twilio docs): GSM-7 fits 160 characters, 153 per part when split; UCS-2 fits 70, then 67. One character outside GSM-7, an emoji or a curly apostrophe, switches the whole message to UCS-2.

Your turn · checkpoint 5not answered

Count the segments

(a) A 95-character English OTP; (b) the same alert in Hindi, 95 characters; (c) a 150-character English payment alert whose template contains one curly apostrophe (’). How many segments each?

Think it through, or jot it down. Then compare.

One strong answer

  • (a) 1: GSM-7, under 160.
  • (b) 2: Devanagari is UCS-2; 95 > 70, so it splits into 67-character parts.
  • (c) 3: the ’ forces UCS-2; 150 ÷ 67 rounds up to 3. With a straight ' it's 1.
  • So one character can triple the cost of every message from a template. Check segments before a template ships.
WorksheetWhat the super-app's SMS costs, and what cuts it (inputs assumed unless marked)
Value
SMS a day (Day 1: 5% of 500M) 25M → 750M a month
Mix (assumed: a platform that hasn't yet applied Day 1's channel rules, so marketing still goes by SMS) 10M / 10M / 5M a day
Segments: OTPs in English, 1 each 10M
Payments: 70% English (1), 30% Hindi and other scripts (2) 7M + 6M = 13M
Marketing: long messages, 2 each 10M
Segments a day 33M
Price per segment (Twilio US list $0.0083 + a $0.0035 carrier fee) ~$0.012
List-price spend ~$396,000 a day, ~$11.9M a month
Lever 1: marketing moves to push, in-app and email digests −10M segments
Lever 2: payment SMS only when push actually failed (assume half today are unnecessary) −6.5M segments
After both 16.5M segments → ~$198,000 a day, ~$5.9M a month

Conclusion: the bill is messages × segments × price, and the platform controls the first two. Halving it needed no negotiation: SMS only for critical sends and real push failures, nothing marketing-shaped. Negotiated rates (Day 1 assumed them) change the totals, not the ratios; India's list price ($0.0832) is about 7× the ~$0.012 US price used here, so the levers matter more there. Every SMS should record why it was sent, so spend per hour can be split by type and alerted on.

5.2 Dedup and Digesting#

The cheapest notification is the one never sent:

  • Dedup at every stage: Day 2's ingestion key, the campaign key, the send ledger. LinkedIn's ATC "handles the duplicated requests by dropping any request it has already processed" (2018).
  • Digest what can wait. ATC can "send a group of notifications together as one notification at a later time"; with it, LinkedIn "cut member complaints in half" (2018).
  • Collapse what replaces itself with apns-collapse-id and FCM collapse keys.
  • Cap frequency per user across teams, as Uber's gateway does (2022).

5.3 The Spend Alert Is a Security Alert#

SMS pumping uses your phone-number fields, usually sign-up or OTP, to send texts to a range of numbers controlled by one mobile operator, and the attacker gets a share of the revenue those texts generate. Twilio's signs: "a spike of messages sent to a block of adjacent numbers", and no "completed verification cycle" (Twilio docs). So alert on spend per hour by country, track OTP conversion (codes entered ÷ codes sent) per country and prefix, rate-limit OTP sends per number, IP and device, and allow only countries you serve.

Chapter 6: Regional Failures#

6.1 What a Region Takes With It#

On 19–20 October 2025, a race condition left an empty DNS record for DynamoDB's endpoint in AWS us-east-1. Impact ran from 11:48 PM to 2:20 PM PDT; "between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed", and Lambda's SQS polling "failed and did not recover automatically" (AWS post-event summary, 2025). Two lessons: you can't scale up in a broken region, so failover capacity must already be running; and unconsumed queues become backlogs that hurt again on recovery.

Your turn · checkpoint 6not answered

21:00, region A goes dark

Region A, home to half the users, fails at 21:00. In flight: a payment alert marked sent in A at 20:59:58, a digest scheduled for 21:00 for users in Tokyo, and 5M open sockets. Replication lags 3 seconds. What happens to each when region B takes over, and what must B do before sending anything?

Think it through, or jot it down. Then compare.

One strong answer

  • Payment alert: its "sent" mark may not have replicated, so B sends it again: a duplicate, collapsed on the device. Better than losing it.
  • Digest: the schedule replicated, so B can fire it, but only after fencing A (bumping a region epoch A's senders check). Otherwise a half-alive A sends it too.
  • 5M sockets: they reconnect into B. Without jitter and admission control that's Day 3's storm at twice the size; with them, B admits everyone in a minute or two.

6.2 A Two-Region Design#

A typical design (ours, not any one company's):

  • Each user has a home region; the outbox, inboxes and send ledger replicate to the other region asynchronously, seconds behind.
  • On region loss, fence first: bump a region epoch so a half-alive region can't keep sending (Week 5, Day 5 · Leader Election).
  • Replay unsent notifications, critical first, bulk last. "Sent" marks from the final seconds didn't replicate, so a few duplicates go out; collapse ids hide them on push.
  • Re-admit sockets with Day 3's admission control; clients resume from the inbox.
  • Provider callbacks must be able to land in either region.
DiagramLosing a region: fence, replay, re-admit
%% title: Losing a region: fence, replay, re-admit
flowchart TD
  subgraph R1["Region A · lost"]
    A1["Pipeline for A's users"]:::bad
  end
  subgraph R2["Region B · survivor"]
    F["Fence<br/>region epoch bumped"]:::core
    RP["Replay unsent from the replicated outbox<br/>send ledger skips sent, critical first"]:::core
    GW@{ icon: "sd:gateway", label: "Gateways<br/>admission control" }
    DB@{ icon: "sd:database", label: "Replicated store<br/>outbox, inboxes, ledger" }
  end
  U@{ icon: "sd:users", label: "A's users<br/>reconnect with jitter" }
  P@{ icon: "sd:cloud", label: "APNs, FCM, SMS<br/>global endpoints" }
  A1 -.->|"async replication, seconds behind"| DB
  F --> RP
  DB --> RP
  RP --> P
  U --> GW
  GW -->|"resume after last seq"| DB
  class GW edge
  class DB store
  class U client
  class P external

Chapter 7: Edge Cases at Scale#

At 50M users, a one-in-a-million case happens 50 times a day.

7.1 Time Zones and Quiet Hours#

"No marketing between 22:00 and 08:00": whose 22:00?

  • Store an IANA zone (Asia/Kolkata), never a fixed UTC offset: a zone's offset changes with daylight saving. Update it from the device when the app opens, since travellers move.
  • Daylight saving makes missing and doubled times. In America/New_York, 02:30 doesn't exist on the spring-forward day and 01:30 happens twice in autumn. A scheduled send must resolve to one real instant.
  • Rules follow the recipient. In the US, for example, telephone solicitations (marketing calls and messages, which includes texts) to residential subscribers are barred "before the hour of 8 a.m. or after 9 p.m. (local time at the called party's location)" (47 CFR 64.1200(c)(1)). The rule covers telemarketing, not push or transactional alerts, and messages sent with the recipient's permission or within an established business relationship fall outside its definition of a solicitation; but where a rule like it applies, a phone number's area code isn't where its owner is.
  • Critical bypasses quiet hours: a 03:00 security alert is the point.
pythonnext_allowed · 16 lines
# routing/quiet_hours.py: next allowed send time, DST-safe (illustrative)
from datetime import datetime, time, timedelta
from zoneinfo import ZoneInfo

QUIET_START, QUIET_END = time(22, 0), time(8, 0)

def next_allowed(now_utc: datetime, tz_name: str, priority: str) -> datetime:
    if priority == "critical":
        return now_utc                                        # security alerts ignore quiet hours
    tz = ZoneInfo(tz_name)
    local = now_utc.astimezone(tz)
    if QUIET_END <= local.time() < QUIET_START:
        return now_utc                                        # outside quiet hours: send now
    day = local.date() if local.time() < QUIET_END else local.date() + timedelta(days=1)
    wake = datetime.combine(day, QUIET_END, tzinfo=tz)        # 08:00 local, on the right date
    return wake.astimezone(ZoneInfo("UTC"))                   # one real instant, even across DST

Campaign waves reuse it: release each zone's recipients when their window opens.

7.2 Uninstalls and Stale Tokens#

Nobody tells you about an uninstall; a send fails, sometimes weeks later.

  • APNs 410 Unregistered carries a timestamp: when "APNs confirmed the token was no longer valid". Delete only if the token wasn't registered again after it.
  • FCM UNREGISTERED (404) and INVALID_ARGUMENT (400) mean "it is safe to delete your record", but INVALID_ARGUMENT "signals an invalid registration only if the payload is completely valid" (Firebase docs).
  • Tokens also go stale silently. By default FCM treats a registration as stale "if its app instance hasn't connected for a month"; on Android, 270 days inactive means expired. Firebase recommends refreshing token timestamps about monthly and pruning stale ones, which otherwise appear as "a dramatic (but not actually valid) drop in delivery rates".
pythonon_push_result · 11 lines
# push/tokens.py: act on what providers say about tokens (illustrative)
def on_push_result(token, r):
    if r.provider == "apns" and r.status == 410:
        if token.registered_at_ms <= r.timestamp_ms:          # dead since our last registration
            tokens.mark_dead(token.id, r.reason)
    elif r.provider == "apns" and r.reason == "BadDeviceToken":
        tokens.mark_dead(token.id, "bad_token")               # often a sandbox token in production
    elif r.provider == "fcm" and r.error == "UNREGISTERED":
        tokens.mark_dead(token.id, "unregistered")
    elif r.provider == "fcm" and r.error == "INVALID_ARGUMENT" and payload_is_valid(r.notification):
        tokens.mark_dead(token.id, "invalid")                 # only then is the token to blame

An uninstall is a choice: fall back to SMS for critical notifications only. And watch invalidation by app version: a doubling on release day means the release broke registration.

7.3 Duplicate Sends Across Retries#

Queues redeliver, workers time out, breakers fail over, regions replay. Exactly-once delivery to a phone isn't available; at-least-once with dedup and collapse is (Week 2, Day 2 · Idempotency in Practice):

Duplicate source What stops it
Producer retries the API Ingestion idempotency key (Day 2)
Campaign job restarts (campaign_id, user_id, channel)
Redelivery after a worker crash Send ledger (notification_id, channel): record "attempting", send, record "sent"
Timeout, then retry or failover Same body and code every attempt; collapse ids
Region replay Ledger, plus collapse ids for what slips through

Graph dedup hits per stage: a jump means a worker is crash-looping or a breaker is flapping.

Part IV · Stage 4 of 5 · 1 min

Part IV: Real-World Application#

Chapter 8: How Real Companies Run It#

  • GOV.UK Notify (2020): an always-split SMS pair, shifted in steps; "between 100,000 and 200,000 text messages a day".
  • LinkedIn ATC (2018): one gateway for email, push and SMS with rate limits, digests, locale-aware timing and dedup; "over a billion requests per day".
  • Uber (2020, 2022): a push platform at 1.5M concurrent connections and 250,000 messages a second with three priorities, and a gateway enforcing per-user caps.
  • Netflix RENO (2022): priority-specific queues and clusters, and a staleness filter.

Chapter 9: Common Mistakes#

  1. Measuring the worker, not the user. "Send took 180 ms" while the message waited 4 minutes. Measure from created_at; watch lane age.
  2. Lanes that end at your border. Separate topics, then one FCM quota and one SMS sender: the OTP waits at the provider. Reserve quota, split senders.
  3. All-or-nothing failover. 100% of traffic onto a cold provider repeats GOV.UK Notify's 2019 failure. Stay split; shift in steps.
  4. Counting bad tokens as provider failures. A token cleanup opens the breaker on a healthy APNs.
  5. No owner for SMS spend. A retry loop or pumping attack shows up on next month's invoice.

Part V · Stage 5 of 5 · 5 min

Part V: Interview Preparation#

Chapter 10: The Whole Week in One Interview#

10.1 The 45 Minutes#

"Design a notification platform" is a common prompt; this week is the full answer.

Minutes What you do Week 6
0–5 Requirements: types, channels, 50M users, 500M a day, 10M an hour in campaigns, latency per class, 99.9% for critical, marketing never delays transactions Day 1
5–10 Estimates: ~5,800 a second average, ~17,000 peak, channel mix, SMS cost, 10M sockets at the evening peak Days 1, 3, 5
10–20 Architecture: API with idempotency keys → outbox → router → lanes → channel workers → providers; inbox and gateways for in-app Days 1, 2
20–35 Deep dive, interviewer's pick: delivery guarantees, real-time delivery and reconnect storms, presence and group fan-out Days 2, 3, 4
35–45 Operations: SLOs per class, lanes to the provider, breakers and SMS failover, cost, regions, edge cases Day 5
DiagramThe week's platform, end to end
%% title: The week's platform, end to end
flowchart TD
  SVC@{ icon: "sd:server", label: "Product services<br/>payments, security, marketing" }
  API["Notification API<br/>idempotency key"]:::core
  OB@{ icon: "sd:database", label: "Notification store and outbox" }
  RT["Router<br/>preferences, quiet hours, caps, dedup"]:::core
  LN@{ icon: "sd:queue", label: "Lanes<br/>critical · transactional · bulk" }
  CW["Channel workers<br/>token buckets, breakers, send ledger"]:::service
  APNS@{ icon: "tech:apple", label: "APNs" }
  FCM@{ icon: "tech:firebase", label: "FCM" }
  SMS@{ icon: "sd:sms", label: "SMS providers A and B<br/>weighted split" }
  EM@{ icon: "tech:sendgrid", label: "Email provider" }
  IB@{ icon: "sd:database", label: "In-app inbox<br/>seq per user" }
  GW@{ icon: "sd:gateway", label: "Gateways<br/>WebSocket and SSE" }
  CH["Chat owners, presence shards<br/>Day 4"]:::service
  DEV@{ icon: "sd:phone", label: "Apps and browsers" }
  OBS@{ icon: "sd:dashboard", label: "SLOs, lane age, spend<br/>alerts and runbooks" }
  SVC --> API --> OB --> RT --> LN --> CW
  CW --> APNS
  CW --> FCM
  CW --> SMS
  CW --> EM
  CW --> IB --> GW
  CH --> GW
  GW <--> DEV
  APNS --> DEV
  FCM --> DEV
  CW -.-> OBS
  class SVC,DEV client
  class OB,IB store
  class LN queue
  class APNS,FCM,SMS,EM external
  class GW edge
  class OBS service
❯ Say this

10.2 Key Phrases#

Phrases that show you've run one of these
MEASURING

"End-to-end from creation time, and the oldest message's age per lane. Success rates can't see a queue."

PROMISING

"SLOs stop at the provider, because APNs doesn't report the device: 99.9% of critical handed off within 10 seconds. That's about 14 minutes of stall at peak a month, so critical gets its own workers and quota."

BURSTS AND PROVIDERS

"Campaigns are paced, pausable jobs in their own lane, and lanes continue to the provider: reserved quota, OTPs on their own SMS sender. SMS is always split across two providers and shifted in steps, never failed over all at once."

10.3 Common Follow-Up Questions#

Question Strong answer
"How do you know a push was delivered?" "Mostly you don't: we measure hand-off, SMS receipts, in-app acks and opens."
"What protects OTPs from a 10M campaign?" "Separate lanes, pools and reserved provider quota; a paced, pausable campaign."
"APNs is down." "Hold with TTLs, in-app if connected, SMS for critical only, ramp the drain."
"How do you avoid sending twice?" "Dedup keys, a send ledger, the same body every attempt, collapse ids."

Ask the interviewer: which notifications are critical, and how late is "late"? One region or several? Is there an SMS budget?

Chapter 11: Practice Problems#

Practice · problem 1

Problem 1: Election-Night Alerts#

Setup: Push "results declared" to 30M users within two minutes, while breaking-news alerts to 200,000 journalists stay under 10 seconds. What do you do the day before, and during?

Click for solution outline
  • 250,000 a second, 25× FCM's default quota: ask Firebase well ahead, precompute the audience, pre-warm workers and APNs connections.
  • Lanes: journalists critical with reserved quota; the mass alert ramped in its own lane, one collapse id per race.

Practice · problem 2

Problem 2: The Silent SMS Provider#

Setup: Provider A returns 200s, but 40% of its receipts to Indian numbers take over 5 minutes. The breaker never opens. Why, and what changes?

Click for solution outline
  • The breaker only counts HTTP status. Add receipt latency as a failure signal, GOV.UK Notify style, scoped per route: "A to India" loses share, "A to the US" keeps it.
  • OTPs get a short validity period, so stuck texts die instead of arriving after the code expires.

Practice · problem 3

Problem 3: The Release That Lost iOS#

Setup: After an iOS release, token invalidation for that version is 30× normal, mostly BadDeviceToken. Diagnose, contain, prevent.

Click for solution outline
  • Diagnosis: the build registers sandbox tokens, which production APNs rejects.
  • Contain: route those users' critical notifications in-app or by SMS, ship a fix, and stop sending to the bad tokens: many BadDeviceToken errors get connections dropped sooner.
  • Prevent: alert on invalidation by app version, and make canaries watch it.

Summary#

DAY 5 KEY TAKEAWAYS
MEASURE AND PROMISE
  • End-to-end from created_at and lane age; SLOs stop at the provider hand-off
BURSTS AND PROVIDERS
  • Paced, pausable campaigns; lanes reach the provider via reserved quota and separate SMS senders
  • Breakers per provider and route; SMS always split, shifted in steps
COST, REGIONS, EDGE CASES
  • SMS = messages × segments × price; cut the first two
  • Regions: fence, replay from the ledger, re-admit slowly
  • IANA zones, careful token deletion, same body on every retry

Further Reading#

Engineering Posts#

Provider Documentation#

this lesson in shorts · 4

play them all →

One idea from this lesson in about a minute each, made for your phone.

End of Day 05 · Operations, Scale and Edge Cases
Finish Day 5Stages clear as you read them (0/5 so far). Skipping ahead doesn't count: marking complete unlocks after ~5 more min of reading.

Go deeper · reading trail

all reading →

Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.

GOV.UK NotifyBlog post · ~6 min

How GOV.UK Notify reliably sends text messages to users

David McDonald · 2020
What
How the UK government's notification service sends 100,000 to 200,000 texts a day through two SMS providers.
Key idea
Failing all traffic over to a cold provider broke it too, so every text now picks a provider at random, about 50/50, and errors or slow receipts shift 10 points at a time.
Why here
The Preface and SMS failover chapter of this lesson: stay split, shift in steps, and never move a whole spike onto a provider that hasn't seen it.
taught here: SMS failover in steps
JioHotstarBlog post · ~5 min

T For Tsunami : Dealing with traffic spikes

Akash Saxena · 2017
What
Hotstar's retrospective on traffic spikes during the 2017 Champions Trophy.
Key idea
A push notification sent as Virat Kohli walked in to bat took login traffic from 10K to 125K TPS in seconds, and the database had not been scaled for it.
Why here
Why this lesson treats a campaign as a paced, planned job: a mass notification is also a load test on everything users land on.
taught here: Campaign mode

Kept from the earlier trail (checked 2026-10-18); blog.hotstar.com answered the 2026-10-29 automated re-check with 403.

Amazon Web ServicesPostmortem · ~20 min

Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region

2025
What
AWS's post-event summary of the October 2025 DynamoDB disruption in us-east-1 and its knock-on effects.
Key idea
A DNS race condition left DynamoDB's regional endpoint with an empty record; new EC2 launches then failed for hours and Lambda's SQS polling had to be restored by hand.
Why here
This lesson's regional-failure chapter: you can't scale up in a broken region, and queues left unconsumed become backlogs that hurt again on recovery.
taught here: Regional failures

New lessons by email

New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.

Discussion · …

Public: everyone reading this lesson sees comments. Be kind and keep it on the material.

Sign in to ask a question or answer one. Reading stays open to everyone.
loading the discussion…