Day 1Designing a Real-Time Notification Platform

Two halves. Store-and-forward through providers (push, SMS, email), where "delivered" mostly means a provider accepted it; and a real-time tier where we hold a connection to online users and deliver ourselves in under a second.

Five classes, different clocks. Critical (OTP, fraud), transactional (payments), social, reminder and marketing each have their own latency, channels and quiet-hours rule, and the lane follows. Marketing must never delay a transaction.

Inbox first, then the hint. In-app delivery writes the item to a durable per-user inbox with a sequence number, then sends the gateway a hint. Lose the hint and the client resumes "after 1183". No ack within seconds: push.

OTPs skip the ladder. A code is useless late, so an OTP goes by SMS at once. Security alerts start with push and escalate to SMS if still unread after 2 minutes. Critical ignores opt-outs and quiet hours.

Sent is not delivered. FCM counts a send once the message is "enqueued for delivery". Keep one deliveries row per channel attempt, separate from the notification and the inbox item, and keep sent and delivered apart.

• One queue for every class: the OTP that leaves at 20:08.
• Treating a provider's 200 as delivery.
• The socket as the record: every crash, dropped Wi-Fi and deploy loses messages.

Day 2The Delivery Pipeline: Guarantees, Priorities and Preferences

Key the business event. The idempotency key names the event (payments:pay_8812:received), not the HTTP request, and commits with the notification and its outbox row. A retry gets the original id; a reused key with different parameters gets 422.

Check late, then again. The router reads cached preferences, but the channel worker re-checks opt-out, quiet hours, expires_at and business conditions just before the provider call, marking failures suppressed or expired.

Lanes all the way down. Critical, transactional and bulk each get their own topic, consumer group, workers, retry topics and share of provider quota. One topic with a priority field can't skip a partition's backlog.

At-least-once, deduplicated. Exactly-once isn't available through a provider you don't control. Claim, send, record per task; dedup at every hop; collapse ids hide the rare duplicate. Bulk treats an unconfirmed claim as sent.

Escalation is a forward-only saga. Fraud alert: push now, SMS if unread at 2 minutes, email 10 minutes later, fraud team at 24 hours. Durable timers, not sleep(120); "undo" cancels the steps not yet taken.

• Promising exactly-once instead of at-least-once with dedup.
• Checking preferences only at routing, so opt-outs lag behind the queue.
• Retrying a 410, or delivering an expired OTP.

Day 3Real-Time Channels: WebSockets and SSE at Scale

Pick the transport per surface. SSE for the one-way web bell: free reconnect and Last-Event-ID resume, and HTTP/2 lifts the 6-connection cap. WebSockets for apps that ack. Long polling as the last resort.

Size by connections. 10M sockets heartbeating every 30 s is ~333,000 beats a second against ~1,700 notifications at peak. Cap nodes at 200,000 for blast radius: 50 nodes, 75 to survive a zone.

Registry with leases and epochs. The registry maps user and device to gateway and epoch. Each gateway holds a 15-second lease; lookups skip dead leases and old epochs. One pub/sub channel per gateway.

The socket is a hint. Redis Pub/Sub is at-most-once, which is fine because the inbox already has the item. Clients resume "after seq N" with a range read; no ack within seconds falls back to push.

Heartbeat under every idle timeout. TCP keepalive waits 2 hours; one carrier NAT cut idle TCP at 255 s; an ALB's default is 60 s. Client heartbeat every 25–30 s with jitter; server closes after ~2.5 misses.

Shape the reconnect storm. Full-jitter backoff with a jittered first attempt, connect admission control answering "retry after", resume instead of reboot, bounded per-socket queues, and slow drains: 1% every 6 seconds.

• Using the socket as the record.
• Unbounded per-connection buffers: a few slow phones run the node out of memory.
• Fixed-delay reconnects that recreate the wave every 2 seconds.

Day 4Presence and Chat Fan-Out at Scale

Three classes of event. Durable messages are stored first, ordered and retried. Presence is soft state in memory with a TTL, latest wins. Typing is ephemeral: throttled, expired by the receiver, dropped first.

Presence expires by itself. Heartbeat every d seconds; the entry expires after d + ε, and a delayed trigger at d + 2ε publishes offline only if no newer beat came. Last seen uses the server's clock.

Tell only who's looking. Clients subscribe to the users on their screen. Eager presence costs ~84,000 pushes a second, twice the message traffic; visible-only with debouncing, ~22,000, about half.

Bodies once, pointers to small groups. The body goes once into the conversation log; small groups get a pointer in each member's inbox; huge rooms get no per-member writes, and full live pushes only reach open sessions.

One owner sequences each conversation. The owner assigns 1, 2, 3…, never the sender's clock. A new owner starts from the durable max, and a unique (conversation_id, seq) key fails a stale owner's write.

Receipts are cursors. "Delivered up to 412" and "read up to 410", applied as max(old, new), so ticks never move backwards. A group's tick is the minimum cursor over its members.

• Writing presence to the message database like messages.
• Broadcasting presence to every contact.
• Ordering messages by the sender's timestamp.

Day 5Operations, Scale and Edge Cases

Measure the user's clock. End-to-end latency from created_at, not dequeue time, and the oldest message's age per lane. Success rates only count attempts, so a four-minute queue looks perfect.

SLOs stop at the provider. We can't see an iPhone show a banner. Critical: 99.9% handed off, or acked in-app, within 10 seconds. Bulk has no availability target: pausing a campaign is the system working.

Lanes reach the provider. FCM's quota is per project and an SMS sender has one queue. Reserve quota that bulk can't touch, use separate SMS senders for OTPs and marketing, and short validity periods.

Shift SMS in steps. GOV.UK Notify's failover moved a spike onto a cold provider, which failed too. Always split about 50/50; a 5xx cuts that provider's share 10 points, at most once a minute.

Same code on every attempt. A timeout may have delivered, so failover can send twice. Create the OTP once and key the ledger (notification_id, channel), not by provider, so a duplicate text is harmless.

• Measuring the worker, not the user.
• Counting bad tokens as provider failures, opening a breaker on a healthy APNs.
• No owner for SMS spend: a retry loop or pumping shows up on next month's invoice.

BossFood Delivery Order System

Size the dinner rush. 5M orders a day is ~58 a second; dinner is 4×, ~230. That's ~7,000 writes and ~17,000 browsing reads a second, and ~400K active orders to track.

Never lose an order. Write the order and an outbox row in one transaction; the publisher marks rows published only after Kafka confirms, so a crash republishes and idempotent consumers absorb the duplicate.

Payments: authorize, capture, refund. Authorize at order time, capture when the driver picks up, refund on cancellation, every Stripe call with an idempotency key, so a retried authorization can't create a second PaymentIntent.

No driver assigned twice. Two orders can both see Driver X as free. Assign under a row lock with a version check, so one succeeds and the other tries its next candidate.

Push live tracking, don't poll. Polling 400K orders every 10 s is ~40,000 mostly empty reads a second. One SSE or WebSocket connection per screen: location coalesced to the latest point, status resumed by sequence number.

Order updates through the lanes. Order updates are transactional: inbox first, a live hint, push only if no device acks. OTPs and driver offers are critical; promotions are bulk. Driver online status is presence, never used for correctness.

• Saving the order, then publishing to Kafka: Kafka down means a held card and no food.
• Retrying payment calls without an idempotency key.
• Reassigning an order because a driver's presence dot went grey.