Week 6 Philosophy#
Unlike Weeks 1-5 where each day covered a new concept, Weeks 6-8 are immersive practical weeks. We take ONE complex real-world system and spend the entire week designing it end-to-end.
This week's system has two halves. One is store and forward through providers: push (APNs, FCM), SMS and email, where we hand the message to someone else's network. The other is real time to online users: when the app is open, we hold a connection and push down it ourselves, in under a second. The same real-time tier also carries chat and the green "online" dots.
Week 6 designs one notification platform with two halves: store and forward through providers (push, SMS, email) and a real-time tier that holds connections to online users. The problem: a fintech super-app in India with 50M users, 500M notifications a day, 10M-an-hour campaigns, 99.9% delivery for critical, and ~10M live connections at the evening peak. Day 1 draws every box, Day 2 builds the delivery pipeline, Day 3 the WebSocket and SSE gateways, Day 4 presence and chat fan-out, Day 5 operations and the interview. Its production example, the zombie socket, is fixed by jittered 25–30 s heartbeats, reconnecting or pushing when no ack comes, and resuming from the inbox.
WEEKS 1-5: Learn concepts
- Day 1 — Topic A
- Day 2 — Topic B
- Day 3 — Topic C
- Day 4 — Topic D
- Day 5 — Topic E
WEEK 6: Apply concepts to ONE real system
A REAL-TIME NOTIFICATION PLATFORM
- Day 1 — Designing a Real-Time Notification Platform
- Day 2 — The Delivery Pipeline: Guarantees, Priorities and Preferences
- Day 3 — Real-Time Channels: WebSockets and SSE at Scale
- Day 4 — Presence and Chat Fan-Out at Scale
- Day 5 — Operations, Scale and Edge Cases
Why a Real-Time Notification Platform?#
A notification platform is the perfect teaching system because it touches EVERYTHING: durable pipelines, external providers you don't control, millions of long-lived connections, soft state that must expire by itself, and a bill that SMS can blow up.
- Week 1 (Data at Scale)
- Partitioning: the inbox by
user_id, chat logs byconversation_id - Rate limiting: frequency caps, connect admission control, provider token buckets
- Hot keys: a 300,000-member chat room during a cricket final
- Week 2 (Failure-First Design)
- Idempotency keys from the business event; client message ids for chat sends
- Timeouts: heartbeats shorter than every idle timeout on the path
- Circuit breakers per provider and route; classified retries
- Distributed cron for scheduled reminders and campaigns
- Week 3 (Messaging & Async)
- Transactional outbox from the Send API to the priority lanes
- Backpressure: a slow client on a socket, a provider that answers 429
- Delay topics and a dead letter queue per channel
- A log per conversation, ordered by sequence number
- Week 4 (Caching)
- Preferences and device tokens cached, invalidated on every change
- Write fan-out vs read fan-out, as in feed caching
- Week 5 (Consistency & Coordination)
- Per-conversation ordering instead of one global order
- The escalation saga: push, then SMS, then email, on durable timers
- Fencing: a unique
(conversation_id, seq)key, a region epoch
The Problem Statement#
You're building the notification infrastructure for a fintech super-app in India: payments, bill pay, and group chats where friends split costs. Users receive notifications about:
- Transactions ("₹2,400 received from Arjun", refunds, bills paid)
- Security (OTPs, new-device logins, fraud alerts)
- Social (chat messages, split requests, mentions)
- Reminders (bill due tomorrow, low balance)
- Marketing (cashback offers, new features)
Channels: push (APNs, FCM), SMS, email, and in-app (a bell with an unread count in the mobile and web apps). No WhatsApp channel for now.
Real time and chat
- Online users see notifications and chat live, in under a second
- Group chats show who's online
- About 15% of users are online at the evening peak: ~7.5M people, ~10M live connections
Scale
- 50M users, about 10M daily active
- 500M notifications/day; campaigns can add 10M in an hour
- 99.9% delivery for critical notifications
Latency
- Security alerts near-instant; transactions within 1–2 seconds; live in-app under 1 second; marketing within a minute
Rules
- Users opt in or out per category and channel, set quiet hours and unsubscribe in one tap; OTPs and security alerts can't be turned off
- SMS is expensive: use it carefully. Local SMS routes in India need the use case and message templates registered with TRAI through DLT
- Marketing must never delay a transaction or security alert, and every send needs an audit trail
Week 6 warm-up
Q1/2 What are the two halves of this week's notification platform?
Answer & explanation
Answer: B
Push, SMS and email are handed to providers; when the app is open, the platform holds a connection and delivers itself in under a second. The same real-time tier carries in-app notifications, chat and presence.
Q2/2 About 15% of the super-app's 50M users are online at the evening peak, with ~1.33 devices each. What should the real-time tier be sized by?
Answer & explanation
Answer: C
7.5M people online with ~1.33 devices each is ~10M sockets; heartbeats alone outnumber in-app notifications more than 100 to 1. So the gateway fleet is sized by connections, memory and reconnect admission, not message throughput.
Week 6 Daily Breakdown#
Day 1: Designing a Real-Time Notification Platform#
Theme: "Draw every box once: requirements, numbers, architecture, data model"
- INTERVIEW APPROACH
- FERNS: functional, edge cases, resources and constraints, non-functional, scale
- The clarifying dialogue, read back as a requirements summary
- Five classes of notification (critical, transactional, social, reminder, marketing), each with its own latency, channels and lane
- ESTIMATES
- 500M a day ≈ 5,800 a second; ~17,400 at a 3× peak; design for 20,000
- Channel mix: push 60%, email 25%, in-app 10%, SMS 5%
- SMS is 5% of messages and ~89% of the provider bill at an assumed $0.004 each
- ~10M live connections at the evening peak
- HIGH-LEVEL ARCHITECTURE
- Seven pieces: ingest, preferences, the pipeline, channel workers, the inbox, the real-time gateways, presence
- One payment notification end to end: inbox write before the socket push, push only if no ack
- The router's policy: OTPs by SMS at once; security alerts escalate to SMS after 2 unread minutes
- DATA MODEL
- Five records: notifications, deliveries, inbox items, preferences with a consent log, device tokens
- Sent is not delivered: a status per channel attempt
- CHANNEL TRADE-OFFS AND RULES
- Push, SMS, email and in-app compared on cost, latency, reach and limits
- Consent per channel, one-click unsubscribe, DLT-registered SMS templates, an audit trail
Week 1-5 concepts applied
- Transactional outbox (Week 3) and idempotency keys (Week 2) at ingest
- Caching preferences (Week 4); partitioning the inbox by user (Week 1)
Day 2: The Delivery Pipeline: Guarantees, Priorities and Preferences#
Theme: "Never lose it, never visibly send it twice, never let a campaign delay an OTP"
- ACCEPTING AN EVENT
- The Send API: an idempotency key named after the business event,
expires_at,collapse_key - The notification and its outbox row in one transaction; cheap, synchronous validation
- The Send API: an idempotency key named after the business event,
- DECIDING WHAT TO SEND
- A preference hierarchy with mandatory categories
- Check at routing, then again at the last mile
- Quiet hours in the user's IANA zone; frequency caps
- Templates pinned by version, with locale fallback
- BATCHING
- Count, window and hybrid flushes; "first at once, then one summary"
- Batching vs collapsing; digests as batches on a schedule
- PRIORITY LANES AND THE INBOX
- Three lanes (critical, transactional, bulk), each with its own topic, workers and provider quota
- Lanes carried into APNs priority, FCM priority and SMS validity periods
- Inbox first: the durable write before any live push
- GUARANTEES AND FAILURE
- At-least-once with dedup at every hop: claim, send, record
- Classified retries, full-jitter backoff cut at
expires_at, delay topics and a DLQ per channel
- TIME AND ORCHESTRATION
- Scheduling in time buckets; per-zone waves, smeared
- The escalation saga for a fraud alert on durable timers
- Sizing the 20:00 peak with Little's law
Week 1-5 concepts applied
- Idempotency (Week 2), outbox, backpressure and dead letters (Week 3)
- Cache invalidation (Week 4), rate limiting (Week 1), distributed cron (Week 2)
- Saga pattern and saga orchestration (Week 5)
Day 3: Real-Time Channels: WebSockets and SSE at Scale#
Theme: "Today we become the provider"
- FOUR WAYS TO PUSH TO AN ONLINE USER
- Short polling, long polling, Server-Sent Events, WebSockets
- What HTTP/2 and HTTP/3 change; choosing per surface
- THE GATEWAY TIER
- A separate, rarely deployed tier that holds the sockets
- What one connection costs; file descriptors and epoll
- Sizing 10M connections: 200,000 per node for blast radius, 75 nodes across three zones
- DELIVERING TO THE RIGHT SOCKET
- The connection registry, kept honest with a lease and an epoch per gateway
- One pub/sub channel per gateway; at-most-once is fine because the inbox is the record
- Per-user sequence numbers, acks and resume from the inbox
- HEARTBEATS AND DEAD CONNECTIONS
- Every box on the path has an idle timeout; zombie sockets
- Jittered heartbeats every 25–30 seconds; network changes on mobile
- WHEN CONNECTIONS FAIL TOGETHER
- Reconnect storms: full-jitter backoff, admission control, resume instead of reboot
- Backpressure on a slow client; graceful drains on deploy
- IN-APP VS MOBILE PUSH, AND SECURITY
- The ack-timeout fallback to push
- One-time connect tickets, Origin checks, in-band reauth, revocation
Week 1-5 concepts applied
- Partitioning and rate limiting (Week 1), timeouts (Week 2), backpressure (Week 3)
Day 4: Presence and Chat Fan-Out at Scale#
Theme: "Push the hint, pull the truth"
- THREE CLASSES OF EVENT
- Durable (messages, read markers), soft state (presence), ephemeral (typing)
- PRESENCE
- Heartbeats with a TTL of d + ε and a delayed offline trigger at d + 2ε
- Per device underneath; last seen stamped by the server
- Visible-only subscriptions; flapping and debouncing
- TYPING INDICATORS
- Throttled by the sender, expired by the receiver, never stored
- MESSAGE FAN-OUT
- Write fan-out vs read fan-out, and the hybrid: bodies once, pointers to small groups
- Huge rooms and hot rooms: one copy per node, passive sessions, relays
- ORDERING AND RECEIPTS
- One owner per conversation assigns sequence numbers; a unique key fences failover
- Receipts as forward-only cursors; a group's tick is the minimum
- OFFLINE, MULTI-DEVICE AND RELIABILITY
- Store-and-forward and sync from per-device cursors
- Dedup on the client message id and on the sequence number
- Split-brain presence, clock skew, partial fan-out failures
Week 1-5 concepts applied
- Feed caching's fan-out choice (Week 4), hot keys and partitioning (Week 1)
- Consistency models and fencing (Week 5), queue vs stream and the outbox idea (Week 3)
Day 5: Operations, Scale and Edge Cases#
Theme: "Run it: measure, promise, survive bursts and outages, and pay for it"
- WHAT TO MEASURE
- The delivery funnel: provider accepted is not delivered
- End-to-end latency from
created_at; the oldest message's age per lane
- SLOs AND PAGING
- SLOs per class that stop at the provider hand-off
- Error budgets: how long the critical lane can stall
- What pages, and a runbook for a burning critical lane
- BURSTS AND PROVIDERS
- Campaign mode: a paced, pausable job, waved by time zone
- Lanes all the way to the provider: reserved quota, separate SMS senders
- Circuit breakers per provider and route; SMS always split and shifted in steps
- When there's no second provider; the same OTP code on every attempt
- COST
- SMS segments (GSM-7 and UCS-2), dedup and digests, SMS pumping
- REGIONS AND EDGE CASES
- Losing a region: fence, replay, re-admit
- Time zones and DST, uninstalls and stale tokens, duplicates across retries
- THE WHOLE WEEK IN ONE INTERVIEW
- A 45-minute plan that walks Days 1-5 as one answer
Week 1-5 concepts applied
- Circuit breakers and idempotency (Week 2), rate limiting (Week 1), backpressure (Week 3)
- Fencing with leader election (Week 5); SLOs and observability preview Week 10
What Makes This Week Different#
1. Interview-Focused Approach#
Each day includes dialogue showing HOW to present in an interview:
Interviewer: "Walk me through how a payment notification gets sent."
You: "The payments service calls the Send API with an idempotency key named after the business event, payments:pay_8812:received, so every retry carries the same key. The API stores the notification and an outbox row in one transaction and returns 202.
The outbox relay publishes it to the transactional lane, which has its own topic, workers and provider quota, so a campaign can't get in front of it. The router reads preferences, and the in-app worker writes the inbox item first, with the next sequence number for the user. Only then does anyone get told: the live socket if she's online, a push if nobody acks..."
2. Production Reality#
We don't just design — we discuss what ACTUALLY breaks:
EDGE CASE: The Zombie Socket
Problem
- An idle TCP connection sends nothing, so neither end notices when a carrier NAT forgets the mapping
- In a 2011 study of 73 carriers, 4 cut idle TCP in 5 minutes or less, one at 255 seconds; Linux's default TCP keepalive waits 2 hours
- The registry still points at the dead socket, and live notifications vanish into it
If we just trust TCP
- The server keeps a "live" socket that can't deliver
- Users see notifications only "when I reopen the app"
Production solution
- Client heartbeats every 25–30 seconds, with jitter, under the shortest idle timeout on the path
- The server closes after ~2.5 missed beats (75 s) and deletes the registry entry
- The client reconnects after one missed pong, and on every network change
- No ack within a few seconds falls back to mobile push
- On reconnect, the client resumes from the inbox after its last sequence number
3. Code for the Parts That Are Easy to Get Wrong#
Short, illustrative implementations you can reason about in an interview:
# Example from Day 2: claim, send, record (illustrative)
async def deliver_once(task, provider, redis, lane_policy):
key = f"sent:{task.key}" # n_77:push:dev_2
if await redis.get(key) == b"done":
return "duplicate-skipped"
claimed = await redis.set(key, "sending", nx=True, ex=lane_policy.claim_ttl_s)
if not claimed:
state = await redis.get(key)
if state == b"done" or lane_policy.at_most_once: # bulk: unknown is good enough
return "duplicate-skipped"
return "retry-later" # someone is mid-send: don't race them
result = await provider.send(task) # the only unrecoverable step
if result.ok:
await redis.set(key, "done", ex=7 * 86400)
else:
await redis.delete(key) # free the claim so a retry can send
return result4. Edge Case Exhaustiveness#
Every edge case you might face in production OR interviews:
- Accepting a Notification
- The caller retries after a lost
202 - An idempotency key reused with a different request
- Unknown template, missing variables, an
expires_atalready past
- The Pipeline
- An OTP queued behind a 10M-message campaign
- A user opts out while the message waits in the bulk lane
- Quiet hours in the user's zone; DST's missing and doubled times
- A worker crashes after the provider said 200
- Provider 429s, dead tokens (APNs 410, FCM
UNREGISTERED) - A push accepted but never delivered; a fraud alert still unread after 2 minutes
- WebSockets and SSE
- Zombie sockets behind a carrier NAT; Wi-Fi to 4G
- A crashed gateway still listed in the connection registry
- A slow client filling its outbound queue
- A proxy that strips
Upgradeheaders - Tokens that expire while the socket stays open; "log out everywhere"
- Reconnect Storms
- A deploy that restarts a third of the gateways at once
- A lost zone: 3.3M sockets reconnecting together
- Presence and Fan-Out
- A dead phone still showing a green dot
- Presence flapping in a lift; a stuck "is typing…"
- A lost send ack, and a retried message
- A fast phone clock that reorders a chat
- A 300,000-member room in its last over
- A conversation owner that crashes mid-fan-out; split-brain presence
- Operations
- APNs down with no second provider
- An SMS provider that answers 200 while receipts stall
- Failing all traffic over onto a cold provider
- An OTP regenerated per attempt; a curly apostrophe that triples SMS cost
- SMS pumping; a lost region; a release that registers sandbox tokens
Concepts Mapping#
Here's how Week 1-5 concepts map to Week 6:
| Week | Concept | Where applied in the platform |
|---|---|---|
| WEEK 1: DATA AT SCALE | Partitioning | Inbox by user, chat log by conversation, lanes partitioned by user_id |
| Rate Limiting | Frequency caps, connect admission, provider token buckets | |
| Hot Keys | Hot chat rooms: relays, passive sessions | |
| WEEK 2: FAILURE-FIRST | Timeouts | Heartbeats under every idle timeout |
| Idempotency | Keys from the business event; client message ids | |
| Circuit Breakers | Per provider and route | |
| Webhook Delivery | Classifying retryable vs permanent provider errors | |
| Distributed Cron | Scheduled reminders and campaign waves | |
| WEEK 3: MESSAGING | Transactional Outbox | Send API to the priority lanes; fan-out driven from the log |
| Backpressure | Slow clients on sockets; providers as slow consumers | |
| Dead Letters | Delay topics and a DLQ per channel | |
| Queue vs Stream | Per-conversation ordered log | |
| WEEK 4: CACHING | Cache Patterns | Preferences and device tokens |
| Invalidation | An opt-out that must bite in seconds | |
| Feed Caching | Write vs read fan-out for chat | |
| WEEK 5: COORDINATION | Consistency Models | Per-conversation order, not a global one |
| Saga | Fraud-alert escalation, forward-only | |
| Saga Orchestration | Durable timers for escalation | |
| Leader Election | Fencing a conversation owner; a region epoch |
Expected Outcomes#
By the end of Week 6, you will:
- Design a real-time notification platform from scratch in 45 minutes
- Estimate it: volume per channel, the SMS bill, live connections
- Keep an OTP ahead of a 10-million-message campaign, all the way to the provider
- Explain at-least-once delivery with dedup, and why exactly-once isn't on offer
- Choose between SSE, WebSockets and long polling, and size a gateway fleet
- Survive reconnect storms, zombie sockets and slow clients
- Build presence as soft state and fan chat out to groups of any size
- Define SLOs, alerts and runbooks; fail over providers without a stampede
- Cut the SMS bill without hurting OTP delivery
Ready to Start?#
Day 1 begins with the whole-system view: the requirements, the numbers, the architecture, the data model and the channel trade-offs.
We'll approach it exactly as you would in an interview — clarifying requirements, estimating scale, and making initial architecture decisions.
Week 6 of the System Design Mastery Series
this lesson in shorts · 2
play them all →One idea from this lesson in about a minute each, made for your phone.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.