Preface#
Saturday night, the super-app's group chat "Goa Trip", six friends. Priya has paid for the villa and posts a split request. Yesterday's connections are all up. Everything on top of them goes wrong within half an hour. (An illustrative scene: the people, times and amounts are made up; each failure is a classic one.)
- 23:31 — Priya sends "Paid ₹48,000 for the villa, ₹8,000 each please". The server stores it, but its acknowledgement is lost on lift Wi-Fi, so her app retries
- 23:31 — Arjun sees the request twice, and pays twice
- 23:33 — Rohan replies "done!". His phone clock is four minutes slow, so on Meera's laptop "done!" sorts above the request
- 23:40 — Kabir's phone died at 23:12, and the gateway node holding his connection was restarted before it published "offline". Priya still sees him online. When he finally flips, his "last seen" says 23:58, taken from his phone's wrong clock
- 23:44 — "Meera is typing…" has been on screen for nine minutes. The "stopped typing" event was dropped
- 23:59 — The "Cricket Night" community, 300,000 members, 40,000 online, reaches the last over. Every message and presence change costs 40,000 deliveries; the chat tier's CPU hits 100%
None of these is a transport bug. What broke runs on top of the connections: who is online, who gets told, in what order, how many times, and what happens to people who weren't there.
Day 4 runs on top of the sockets. An illustrative night in "Goa Trip" shows a lost ack and a double payment, a fast clock, a dead phone still online, a stuck typing indicator and a 300,000-member room at 100% CPU; events split into durable, soft state and ephemeral. Presence uses heartbeats with a TTL of d + ε and an offline trigger at d + 2ε, per device, with a server-stamped last seen, and visible-only subscriptions cut ~84,000 pushes a second to ~22,000 (assumed traffic). Messages are accepted by one owner per conversation (seq 412) and fanned out as bodies once with pointers to small groups, one copy per gateway node in hot rooms; receipts are forward-only cursors. Sync resumes from per-device cursors, sends and receives are deduped, and failures heal through TTLs and the log.
Yesterday (Day 3) built the pipes: gateways, a connection registry, one pub/sub channel per gateway, a per-user inbox that clients resume from, and the heartbeats, backpressure and reconnect handling that keep it all up. Today assumes they work and builds what runs through them.
Part I: Foundations#
Chapter 1: What Runs on Top of a Connection#
1.1 The Simple Definitions#
- Presence: the system's best guess at whether a person is reachable now (online, idle, offline), plus last seen when they're not.
- Fan-out: turning one event into one delivery per interested device.
- Ordering: every member of a conversation sees its messages in the same order.
- Receipts: how far a message got: stored, delivered to a device, read by a person.
- Store-and-forward: what a device missed waits in an inbox until it reconnects and syncs.
1.2 Three Classes of Event#
These events are not the same kind of thing. Treat them alike and you get presence that's too expensive and messages that are too fragile.
| Durable | Soft state | Ephemeral | |
|---|---|---|---|
| Examples | Messages, edits, membership, read markers | Presence, last seen | Typing, "recording audio" |
| If one is lost | A bug: someone misses a payment request | Heals at the next heartbeat | Nobody notices |
| Stored? | Yes, before anyone is told | In memory, with a TTL | No |
| Acked and retried? | Yes, end to end | No: the next update replaces it | No |
| Ordered? | Per conversation, by sequence number | Latest wins, per user | No |
| Under overload | Queue it, never drop | Coalesce to the newest state | Drop first |
Slack draws the same line: its transient events "are not persisted in the database and are sent through a slightly different flow", and typing is its example (Slack Engineering, 2023).
Chapter 2: Presence#
2.1 Heartbeats With a TTL#
Yesterday's heartbeat asked "is this socket alive?", and the connection registry answered "which gateway holds it?": routing state. Presence asks "is this person reachable?", and it has an audience. Each device tells the presence service it's still there every d seconds, and the service stores that with an expiry a little longer than d. If nothing renews it, the user goes offline. Nobody has to say goodbye, which matters because phones die, lose signal and get killed by the OS in silence.
LinkedIn published this design for its green dots (LinkedIn Engineering, 2018): a heartbeat every d seconds, an entry that expires after d + ε, and a delayed trigger after d + 2ε that publishes "offline" only if no newer heartbeat came. A member who "drops their connection and then reconnects within d seconds" stays online, which "prevents unnecessary fluctuations". That's the fix for Kabir: whatever crashed, he goes offline by himself.
%% title: Presence: heartbeats in, visible-only fan-out out
flowchart TD
P1@{ icon: "sd:phone", label: "Kabir's phone<br/>heartbeat every d s" }
P2@{ icon: "sd:laptop", label: "Kabir's laptop<br/>lid closed, entry expired" }
GW@{ icon: "sd:gateway", label: "Gateways<br/>from yesterday" }
PS["Presence shard<br/>owns Kabir · hash of user id<br/>per-device entries, TTL d + ε"]:::core
SUB@{ icon: "sd:cache", label: "Who is watching Kabir<br/>subscriptions from open screens" }
W1@{ icon: "sd:phone", label: "Priya<br/>Goa Trip open" }
W2@{ icon: "sd:phone", label: "Meera<br/>chat list visible" }
X["Kabir's other 148 contacts<br/>not looking: told nothing"]:::service
P1 --> GW
P2 --> GW
GW ==>|"heartbeat"| PS
PS -->|"state changed?"| SUB
SUB ==>|"one event each"| W1
SUB ==>|"one event each"| W2
SUB -.->|"skipped"| X
class P1,P2,W1,W2 client
class GW edge
class SUB cache2.2 Per Device or Per User?#
Store presence per device, and derive the user's state as the most present device: online if any device is active, idle if connected but inactive, offline when the last device expires. Slack's docs use this rule: a user is active "if they have at least one client connected to Slack, and they are not marked as 'away'", and "after 10 minutes with no activity, the user is automatically marked as away". Per-device state keeps a user online on the laptop when the phone dies, and tells delivery (Chapter 6) which devices are reachable.
Publish only when the derived user state changes. A laptop going idle while the phone is active changes nothing anyone sees, so nothing is sent.
2.3 Last Seen#
Last seen is the time of the user's last heartbeat. Stamp it with the presence server's clock when the heartbeat arrives, never the phone's (Chapter 7.2), and record the last heartbeat, not the moment the offline trigger fired. It's usually a privacy setting too, so it's checked against the viewer's permissions on read.
import time
ONLINE, IDLE, OFFLINE = 2, 1, 0
D, EPS = 30, 5 # heartbeat every 30 s, 5 s of slack (illustrative values)
class PresenceShard:
"""Presence for the users hashed to this shard. Memory only: it's soft state."""
def __init__(self, publish):
self.devices = {} # user -> {device: (state, expires_at)}
self.last_beat = {} # user -> server wall-clock time of the last heartbeat
self.shown = {} # user -> state last published to watchers
self.publish = publish
def heartbeat(self, user, device, state=ONLINE):
self.devices.setdefault(user, {})[device] = (state, time.monotonic() + D + EPS)
self.last_beat[user] = time.time() # the server's clock, never the phone's
self._maybe_publish(user)
def sweep(self):
"""The delayed trigger: runs every few seconds, drops devices silent for d + 2ε."""
now = time.monotonic()
for user, devs in list(self.devices.items()):
for dev, (_, expires_at) in list(devs.items()):
if now > expires_at + EPS:
del devs[dev]
self._maybe_publish(user)
def _maybe_publish(self, user):
devs = self.devices.get(user, {})
state = max((s for s, _ in devs.values()), default=OFFLINE) # any device online wins
if state != self.shown.get(user, OFFLINE): # publish changes only
self.shown[user] = state
last_seen = self.last_beat.get(user) if state == OFFLINE else None
self.publish(user, state, last_seen) # to watchers only (2.4)2.4 Who Gets Told#
Knowing presence is cheap; telling people is the expensive part. The naive design publishes every change to every contact. Estimate it before reading on.
Before the numbers: presence for 5 million chat users online
The super-app has 50 million users, and 7.5 million are online at the evening peak (Day 1). Assume 5 million of those are using chat, the only users whose presence is tracked and shown; the rest are paying bills or checking balances. Each chat user's state changes 4 times an hour after smoothing out flaps, and each has 150 contacts on the app, 10% of them online in chat. If every change goes to every online contact, how many pushes a second is that? Compare it with messages: the 10 million daily actives send 45 a day on average, peak is 3× average, and each message reaches 2.5 online devices.
One strong answer
- Changes: 5M × 4 ÷ 3,600 ≈ 5,600 a second.
- Eager presence: 5,600 × 150 = 840,000 lookups a second; 10% online → ~84,000 pushes a second.
- Messages: 450M a day ÷ 86,400 ≈ 5,200/s, ×3 ≈ 15,600/s, × 2.5 ≈ 39,000 deliveries a second.
- So eager presence sends twice the product's message traffic, for a dot most recipients aren't looking at.
The fix is lazy, visible-only presence: a client subscribes to the users on its screen (the open chat, the visible part of the chat list) and unsubscribes as they scroll away. The published systems all do this:
- Slack: "A Slack client receives presence notifications only for a subset of users that are visible in the app screen at any moment" (2023). Moving presence to subscriptions cut presence events reaching clients "by a factor of 5" (2017); the API has required subscriptions since January 2018 and calls "500 users ... a good maximum".
- Discord: sends "the updates for the visible portion of the Member List" only, for "less network traffic, less CPU usage, better battery life" (2019).
- LinkedIn: publishes each member's status on that member's own topic, which reaches the connections "who could be currently viewing the member's presence status" (2018).
| Value | |
|---|---|
| Users / online at the evening peak (Day 1: 15%) / of those, in chat with presence tracked (assumed) | 50 million / 7.5 million / 5 million |
| State changes per online user after debouncing (assumed) | 4 an hour → ~5,600 a second |
| Eager: × 150 contacts, delivered to the 10% online in chat | 840,000 lookups → ~84,000 pushes a second |
| Visible-only: online users watching each changing user (assumed) | 4 → ~22,000 pushes a second |
| Visible-only without debouncing (30 raw flips an hour, assumed) | ~42,000 changes → ~167,000 pushes a second |
| Message deliveries at peak (10M daily actives × 45 a day, ×3, × 2.5 devices) | ~39,000 a second |
| Memory: 5M users × 1.3 devices × ~100 B, plus 100M subscriptions (~20 per online chat user, assumed) × ~32 B | ~0.7 GB + ~3.2 GB |
Conclusion: eager presence costs twice the message traffic; visible-only presence with debouncing costs about half of it, and all its state fits in memory on a few shards. Presence gets its own in-memory, user-sharded service and its own path to the gateways, and is never written to the message database.
Subscriptions are soft state too: after a shard restart, clients re-send their lists (Slack's presence_sub takes the "entire list of user IDs ... in each request") and heartbeats rebuild the states within d seconds.
2.5 Flapping and Debouncing#
A phone in a lift goes online, offline, online in ten seconds, and every flip fans out. Three dampers: the ε in d + 2ε absorbs short drops; delay downgrades, not upgrades (online shows at once, offline can wait a few seconds in case it reverses); and coalesce per watcher, replacing an unsent presence event for Kabir instead of queueing a second. Only the newest state matters, which is why presence can be dropped under backpressure. Slack's API has batched presence changes into one event, with a users array, since November 2017.
Chapter 3: Typing Indicators#
"Meera is typing…" is worth something for a few seconds and nothing after. A typical design (the values are illustrative):
- The sender throttles: at most one
typingevent every 3 seconds while the user types. - The receiver expires: the indicator disappears 5 seconds after the last event, whether or not a "stopped" event arrives. That alone fixes Meera's stuck indicator.
- The server only forwards: no storage, no sequence number, no ack, no retry, and only to members with the conversation open.
- Overload drops it first, and big rooms show "3 people are typing" or nothing at all.
What breaks if typing goes through the message path?
A teammate suggests sending typing indicators through the message pipeline, so they get sequence numbers, durable storage and retries "for free". Each message takes about 10 seconds to type, so a throttled client sends ~3 typing events per message. What happens to storage, ordering and the user's experience?
One strong answer
- Volume: ~3 typing events per message roughly quadruples durable writes, for data that's worthless five seconds later.
- Ordering: typing events take sequence numbers, so a gap-detecting client (Chapter 5) stalls or re-fetches a lost "typing" before showing the next real message.
- Retries: a typing event retried 20 seconds late shows "Meera is typing…" after she's gone.
- Sync: a phone back after a day replays thousands of stale typing events. Ephemeral events must be droppable; the message path is built never to drop.
Part II: Message Fan-Out#
Chapter 4: 1:1, Groups and Huge Rooms#
4.1 One Message, End to End#
Sending has two halves that fail differently. Accept: the device sends the message with its own client message id; the conversation's owner assigns the next sequence number, writes it to the durable log, and only then acks (one tick). Distribute: the message, or a pointer to it, goes to every recipient device, live if connected, through the inbox and sync if not; each device acks once it has stored it (delivered).
%% title: Priya's split request, from her phone to Arjun's sequenceDiagram autonumber participant PP as Priya's phone participant CS as Chat service, owner of Goa Trip participant LOG as Conversation log participant IN as Members' inboxes participant GW as Arjun's gateway participant AP as Arjun's phone PP->>CS: send, client id c-77, "Paid 48,000..." CS->>LOG: append as seq 412, unique on conversation and seq LOG-->>CS: stored CS-->>PP: ack seq 412, one tick CS->>IN: pointer, Goa Trip up to 412, one per member CS->>GW: live copy for connected devices GW->>AP: message seq 412 AP-->>CS: delivered up to 412 CS-->>PP: delivered, two ticks once all members have it
Nothing is acknowledged before it's durable, and nothing is distributed before it's stored. The Discord side quest is designed the same way: writes are stored before anyone is told, so a dropped connection loses the live copy of a message, never the message.
4.2 Write Fan-Out vs Read Fan-Out#
It's the choice from Week 4, Day 4 · Feed Caching: do the work when the message is written, or when it's read?
- Write fan-out (inbox per member): on send, append an entry to every member's inbox; a device syncs by reading one inbox from its cursor. Messenger's Iris works this way: "a totally ordered queue of messaging updates" per user "with separate pointers into the queue", each advanced when an update reaches its destination (Meta Engineering, 2014).
- Read fan-out (log per conversation): write the message once; members read each conversation's log from their own cursor. Discord stores messages by channel, keyed
((channel_id, bucket), message_id), and loads a channel by querying its time buckets from the newest back until it has enough messages (Discord Engineering, 2017).
Write fan-out · inbox per member
- Send: N inbox writes
- Sync: one inbox, one cursor
- Unread counts: fall out of the inbox
- Breaks at: big groups: 300,000 writes per message
- Fits: 1:1 and small groups
Read fan-out · one log per conversation
- Send: one append
- Sync: one query per conversation the user is in
- Unread counts: latest seq minus the member's read cursor
- Breaks at: users in hundreds of quiet conversations, all asked "anything new?" on reconnect
- Fits: large groups and channels
The usual answer is a hybrid: the body goes once into the conversation log, and small groups get a tiny pointer ("Goa Trip is now at 412") in each member's inbox, the same per-user inbox, with its per-user sequence, that yesterday's clients resume from. Sync reads the inbox to learn which conversations changed, then reads only those logs. Huge rooms get no per-member writes at all: members pull when they open the room, and badges compare cursors. Discord's Read States service, which keeps "track of which channels and messages you have read", is "accessed every time you connect to Discord, every time a message is sent and every time a message is read" (Discord Engineering, 2020).
%% title: Hybrid fan-out: bodies once, pointers to small groups, nothing per member in huge rooms
flowchart TD
M["New message<br/>conversation, seq, body"]:::core
LOG@{ icon: "sd:database", label: "Conversation log<br/>key: conversation id, seq" }
Q{"Members?"}
SMALL@{ icon: "sd:queue", label: "Member inboxes<br/>pointer: conversation, latest seq" }
BIG["No per-member writes<br/>unread = latest seq − read cursor"]:::service
LIVE["Live push<br/>only to sessions with the room open"]:::service
GW@{ icon: "sd:gateway", label: "Gateways" }
M ==> LOG
LOG --> Q
Q -->|"up to ~1,000, illustrative"| SMALL
Q -->|"more"| BIG
LOG --> LIVE --> GW
SMALL -.->|"sync reads"| LOG
class LOG store
class SMALL queue
class GW edgePick the fan-out for three conversations
(a) Priya and Arjun's 1:1 chat; (b) "Goa Trip", 6 members; (c) "Cricket Night", 300,000 members, 40,000 online, 5,000 with it open, 2 messages a second in the last over. For each, pick write fan-out, read fan-out or the hybrid, and say who gets a live push.
One strong answer
- (a) and (b): hybrid: the body once, a pointer into 2 or 6 inboxes. A reconnecting phone finds them in its own inbox. Live push to every connected device of every member.
- (c): read fan-out. Pointers would be 300,000 × 2 = 600,000 inbox writes a second for one room. Store once; members pull when they open it.
- (c) live: full messages only to the 5,000 open sessions. The other 35,000 get at most a batched unread bump.
4.3 Huge Groups and Hot Rooms#
Big rooms cost members × activity, and activity grows with members. Discord: "the amount of work needed to handle a discord server grows quadratically with the size of the server"; 1,000 people online each saying one thing is 1 million notifications, 100,000 is 10 billion (Discord Engineering, 2023). In 2017, with servers of "up to 30,000 concurrent users", publishing one event from a large server took "900ms to 2.1s" at peak.
A hot room is a hot key (Week 1, Day 4 · Hot Keys and Skew), and the chat versions of the fixes are:
- One copy per server, not per session. Yesterday's one channel per gateway already gives this for the live path. Discord's Manifold sends to the remote nodes, so the sender calls send no more often than "the number of involved remote nodes" (2017). Slack's channel servers send each message to every subscribed gateway server, and "each GS that receives that message sends it to every connected client subscribed to that channel id" (2023).
- Push only to people looking. Discord stopped sending passive sessions everything from a server until the user clicks in; about 90% of connections in large servers were passive, making fan-out "90% less expensive" (2023).
- Split the room's work. Discord's relays do "fanout with permission checks" for up to 15,000 sessions each, so a server with "over 10 million members, of whom over 1 million are online" keeps working (2023).
- Thin the events: no typing or member presence in big rooms, aggregated reactions, slow mode.
- Isolate it, so a cricket final doesn't slow Priya's 1:1 chat on the same shard.
%% title: Cricket Night: one copy per gateway node, full messages only to people looking
flowchart TD
OWN["Room owner<br/>Cricket Night · seq 88,104"]:::hot
R1["Relay 1<br/>~15,000 sessions"]:::service
R2["Relay 2<br/>~15,000 sessions"]:::service
R3["Relay 3<br/>~10,000 sessions"]:::service
G1@{ icon: "sd:server", label: "Gateway node A<br/>one copy in" }
G2@{ icon: "sd:server", label: "Gateway node B<br/>one copy in" }
ACT@{ icon: "sd:users", label: "Active sessions<br/>room open · full message" }
PAS@{ icon: "sd:users", label: "Passive sessions<br/>badge bump, batched" }
OWN ==> R1
OWN ==> R2
OWN ==> R3
R1 ==> G1
R1 ==> G2
G1 ==> ACT
G2 -.-> PAS
class G1,G2 edge
class ACT,PAS clientChapter 5: Ordering and Receipts#
5.1 Per-Conversation Sequence Numbers#
Rohan's reply jumped the queue because clients sorted by the sender's clock. Yesterday's inbox numbered events per user, which orders one person's notifications. Chat needs a number every member agrees on, so give every conversation one owner that assigns sequence numbers:
- Partition by
conversation_id(Week 1, Day 1 · Partitioning), so one owner hands out 1, 2, 3… and every device shows that order. - Gaps are visible: a device holding 410 that receives 412 fetches 411.
- Order is per conversation only: on the spectrum in Week 5, Day 1 · Consistency Models that's far weaker than one global order, and it's exactly what a log partitioned by conversation gives you (Week 3, Day 1 · Queue vs Stream). A total order would need one global sequencer for no visible benefit.
Ids can still be time-sortable for storage (Discord's Snowflakes are "chronologically sortable", 2017); what users see is fixed by one owner, not by senders' clocks. When the owner fails over, the new one starts from the durable max, and a unique key on (conversation_id, seq) makes a slow old owner's write fail: the key is the fence, as with fencing tokens in Week 5, Day 5 · Leader Election.
class ConversationOwner:
"""Owns some conversations (by hash of conversation_id). One writer per conversation."""
def __init__(self, log):
self.log = log # durable; unique key (conversation_id, seq); also stores client ids
self.next_seq = {} # in memory, rebuilt from the log after a failover
def append(self, conv_id, sender_device, client_msg_id, body):
prior = self.log.find_client_id(conv_id, sender_device, client_msg_id)
if prior is not None: # a retried send: same message, same seq, same ack
return prior
if conv_id not in self.next_seq: # new owner: trust the log, not memory
self.next_seq[conv_id] = self.log.max_seq(conv_id) + 1
seq = self.next_seq[conv_id]
self.log.insert(conv_id, seq, sender_device, client_msg_id, body) # raises if seq is taken
self.next_seq[conv_id] = seq + 1
return seq # ack to the sender only after the insertEach device keeps the highest sequence it has shown contiguously, drops duplicates, holds early arrivals briefly and fetches holes:
class ConversationView:
def __init__(self, shown_upto=0):
self.shown_upto = shown_upto # every seq <= this is on screen
self.early = {} # seq -> message that arrived ahead of a gap
def receive(self, msg, fetch_range):
if msg.seq <= self.shown_upto or msg.seq in self.early:
return [] # duplicate from at-least-once delivery: drop it
self.early[msg.seq] = msg
shown = []
while self.shown_upto + 1 in self.early:
self.shown_upto += 1
shown.append(self.early.pop(self.shown_upto))
if self.early: # a hole: ask for it rather than wait for a retry
fetch_range(self.shown_upto + 1, min(self.early) - 1)
return shown5.2 Delivery vs Read Receipts#
| Receipt | Means | Reported by |
|---|---|---|
| Sent (one tick) | Stored durably | The owner, after the log write |
| Delivered (two ticks) | A recipient device has stored it | That device, after persisting it |
| Read (blue) | A person has seen it | The app, when the conversation is on screen |
WhatsApp's help centre (2022 version) adds the group rule: the second tick appears when every participant has received the message, and the blue ticks when every participant has read it; read receipts are always sent in groups, even if you've turned them off.
Two choices keep receipts cheap:
- Receipts are cursors, not per-message rows. "Delivered up to 412" and "read up to 410": one small event for any number of messages, one row per member per conversation.
- Cursors only move forward. Apply
max(old, new), so late, duplicated or reordered receipts can't move a tick backwards.
A group's tick is then the minimum cursor over members. WhatsApp caps groups at 1,024 (WhatsApp, 2022), which keeps that and the "message info" screen cheap; in a 300,000-member room, per-member receipts are meaningless. Read state gets hot: Discord saw "tens of thousands of database writes per second" and schedules "a database commit for 30 seconds in the future whenever a Read State is updated" (2020), so repeated updates to one cursor become one write.
%% title: A message's status as the sender sees it: it only ever moves forward stateDiagram-v2 [*] --> Sending: tapped send Sending --> Sending: no ack, retry same client id Sending --> Sent: owner stored it Sent --> Delivered: every member has it on a device Delivered --> Read: every member read cursor past it Sent --> Read: read cursor arrives first Read --> [*]
Part III: Offline, Multi-Device and Reliability#
Chapter 6: Store-and-Forward and Sync#
6.1 The Inbox and Sync on Reconnect#
Most recipients aren't connected when a message is sent. WhatsApp's privacy policy: "If a message cannot be delivered immediately (for example, if the recipient is offline), we keep it in encrypted form on our servers for up to 30 days as we try to deliver it", and "Once your messages are delivered, they are deleted from our servers." Discord and Messenger keep history permanently and Slack does on paid plans (its free plan hides messages after 90 days), but catching up works the same way: a cursor per device, and a sync that sends everything after it.
Yesterday's resume was a range query on the per-user inbox; for chat, that query returns pointers, and the logs supply the bodies. The device sends its cursors, the server streams what changed, oldest first, and each ack advances the cursor and drives the senders' "delivered" ticks. After a long absence, send the newest page of each conversation first and load older history on scroll. Messenger calls this "a push-based snapshot + delta model", one snapshot then pushed deltas, and its new sync protocol cut non-media data use by "about 40%" (Meta Engineering, 2014).
%% title: Kabir's phone comes back after 40 minutes offline sequenceDiagram autonumber participant KP as Kabir's phone participant SY as Sync service participant IN as Kabir's inbox participant LOG as Conversation logs KP->>SY: resume, inbox cursor 9,120 SY->>IN: entries after 9,120 IN-->>SY: Goa Trip up to 415, Mom up to 88 SY->>LOG: Goa Trip after 409, Mom after 86 LOG-->>SY: 6 messages and 2 messages SY-->>KP: batch, oldest first KP->>KP: store, drop any already held KP-->>SY: ack, inbox cursor 9,122 SY-->>KP: live from here
def sync(device, inbox, logs, page=200):
"""Everything this device missed, oldest first, a bounded page per conversation."""
changes = inbox.read_after(device.inbox_cursor) # pointers: (conv_id, latest_seq)
batch = []
for conv_id, latest in changes:
have = device.cursors.get(conv_id, 0)
if latest - have > page: # far behind: newest page now,
batch.append(("older_available", conv_id, have)) # the rest loads on scroll
have = latest - page
batch.extend(logs.read(conv_id, after=have, upto=latest))
return batch # the device stores it, then acks: cursors advance, senders get "delivered"6.2 Multi-Device Sync#
- Cursors are per device. The laptop can be at 415 while a tablet off for a week is at 302.
- The sender's other devices are recipients. WhatsApp's client encrypts each message "N number of times to N number of different devices — those in the sender and receiver's device lists" (Meta Engineering, 2021).
- Read state syncs across devices. Reading on the laptop clears the phone's badge and pending notification: a read cursor is a tiny durable event, fanned out to the user's other devices.
- A new device needs history. WhatsApp's phone sends a new device "a bundle of the messages from recent chats", app state is synced through encrypted copies on the server, and a phone can link "up to four other nonphone devices" (2021).
6.3 At-Least-Once Delivery With Client Dedup#
Every hop retries: sends, pushes, sync batches. Exactly-once delivery isn't available over an unreliable network; at-least-once plus idempotent processing is (Week 2, Day 2 · Idempotency in Practice, and Day 2 of this week for the notification pipeline). Chat needs two dedup points: the owner dedups sends on the client message id before assigning a sequence number (find_client_id in 5.1), and devices dedup receives on the sequence number (ConversationView).
The ack that never arrived
The owner stores Priya's request as seq 412, but the ack is lost and her phone retries after 5 seconds. Separately, Arjun's phone stores 412 from the live push, its "delivered" ack is lost, and sync re-sends 412. With no dedup anywhere, what does everyone see? Which two checks fix it, keyed on what?
One strong answer
- Without dedup: the retry becomes seq 413, so everyone sees the request twice (Arjun pays twice), and Arjun's phone may show 412 twice too.
- On send: dedup on (sender device, client message id) before a seq is assigned, remembered for as long as a phone might retry (days). The retry gets "412" back.
- On receive: dedup on (conversation, seq): 412 is already held, so it's dropped and re-acked.
- Not timestamps or text: two identical "ok" messages a second apart are both real.
Chapter 7: Reliability Edge Cases#
7.1 Split-Brain Presence#
A partition leaves two presence nodes each believing they own Kabir. Half his friends see him online, half offline. Don't fix this with consensus on every heartbeat, for a green dot. Instead:
- Let it heal. Presence is soft state with a TTL; once ownership settles, the next heartbeat is the truth and the wrong node's entry expires.
- Version changes per user. The owner stamps each change with an increasing version (ownership epoch in the high bits); watchers keep only the highest, so a stale "offline" can't overwrite a newer "online".
- Never use presence for correctness. Never skip storing a message or a push because presence said "online". A wrong dot is cosmetic; a dropped message isn't.
7.2 Clock Skew in Last Seen#
Kabir's "last seen 23:58" came from his phone's clock. Stamp last seen on the server; keep it monotonic per user (max(stored, new)), so moving between presence servers with slightly different clocks can't make it go backwards; show it coarsely ("today at 23:12") and never in the future; and never order messages by it, or by any phone timestamp.
7.3 Partial Fan-Out Failures#
The owner stores seq 412, then pushes to 40 gateway nodes and 6 inboxes. Node 17 times out, one inbox write fails, or the owner crashes halfway. At scale, this happens daily.
- The log write is the commit point. Once 412 is in the log, the message exists.
- Retry per recipient, not per message; receive-side dedup absorbs any repeats.
- Drive fan-out from the log. A worker consumes the log from a checkpoint, so a crashed owner's unfinished fan-out resumes where it stopped: the idea behind Week 3, Day 2 · Transactional Outbox.
- Sync is the safety net. A device that missed 412 has its cursor at 411; the next sync, or the gap when 413 arrives, fetches it.
- Read membership as of the message. If Rohan leaves at seq 413, a slow fan-out of 412 still reaches him and 414 doesn't.
Partition, skew and a half-finished fan-out
The presence tier splits for 90 seconds during a deploy; Meera's laptop and phone are on opposite sides. Meanwhile the Goa Trip owner stores seq 420 and crashes after pushing it to 3 of the 6 members. What does each friend see, during and after, and which mechanism repairs each piece?
One strong answer
- Presence: some friends may see Meera offline during the split. Afterwards, heartbeats re-establish her within d, and per-user versions stop a stale "offline" winning. Nothing durable depended on it.
- Message 420: it's in the log, so it exists. The fan-out worker resumes from its checkpoint for the other 3; failing that, their devices fetch 420 on the next sync or when 421 reveals the gap.
- Ticks: Priya's two ticks appear once all 6 delivered cursors pass 420. Cursors only move forward, so retries can't confuse them.
| Failure | Users see | Contained by |
|---|---|---|
| Device dies silently | "Online" for d + 2ε | Heartbeat TTL |
| Presence shard restarts | Blank dots for seconds | Clients re-send subscriptions |
| Presence split-brain | Mixed dots for one heartbeat | TTL, per-user versions |
| Send ack lost | Spinner, then a retry | Dedup on client message id |
| Live push lost | Message a little later | Gap check, sync from the cursor |
| Owner crashes mid-fan-out | Some members get it late | Log commit, fan-out checkpoint |
| Owner fails over | Short pause in one chat | Durable max, unique (conversation, seq) |
| Hot room | Slow delivery in that room | Per-node copies, passive sessions, relays |
Part IV: Real-World Application#
Chapter 8: How Real Companies Do It#
| Company | What it has published |
|---|---|
| LinkedIn (2018) | Heartbeats every d s, entries expiring after d + ε, one Akka actor per online member as a delayed trigger at d + 2ε; presence on per-member topics, reaching connections who may be viewing it; millions online, a status change reaches a viewer in under 200 ms at p99 |
| Slack (2017, 2023, API docs) | In-memory presence servers with users hashed to them; presence only for users visible on screen; subscriptions cut presence events 5×; channel servers by consistent hashing send one copy per gateway server; transient typing events skip the database; delivery "across the world in 500ms" |
| Discord (2017–2023) | One process per server routing everything; quadratic fan-out; Manifold one copy per node; visible-portion member lists; passive sessions (90% of large-server connections); relays of 15,000 sessions; channel-keyed storage with Snowflake ids; Read States with 30-second commits |
| WhatsApp (policy, help centre, 2021, 2022) | Undelivered messages kept encrypted up to 30 days, delivered ones deleted; group ticks only when all participants have received or read; groups up to 1,024; per-device encryption, history bundles and up to 4 linked devices |
| Messenger (2014) | Iris: a totally ordered per-user queue with a pointer per destination; snapshot then pushed deltas; ~40% less non-media data |
Chapter 9: Common Mistakes to Avoid#
9.1 Mistake 1: Presence on the Message Path#
WRONG: presence as durable data
- Every change written, replicated and retried like a message
- A reconnect storm becomes a database write storm
- A presence surge delays real messages
CORRECT: presence as soft state
- In memory, sharded by user, with TTLs
- Its own path; coalesced or dropped under load
- Only last seen is persisted, lazily
9.2 Mistake 2: Telling Everyone#
WRONG: broadcast to all contacts
- ~84,000 pushes a second in our estimate, twice the messages
- Most recipients aren't looking
CORRECT: visible-only subscriptions
- Subscribe to who is on screen; debounce and coalesce
- No member presence at all in huge rooms
9.3 Mistake 3: Ordering by Timestamp#
WRONG: sort by the sender's clock
- A slow clock puts a reply above its question
- Devices can disagree, and gaps are invisible
CORRECT: per-conversation sequence numbers
- One owner assigns 1, 2, 3…; every device agrees
- Gaps are detected and fetched; a unique key fences failover
9.4 Mistake 4: Treating the Live Push as Delivery#
WRONG: pushed to the socket = done
- A half-failed fan-out goes unnoticed
- A device reconnecting at that moment never gets it
CORRECT: log, then hint, then cursor
- The log write commits; devices ack what they've stored
- Sync and gap checks repair every missed push
Part V: Interview Preparation#
Chapter 10: Interview Tips and Phrases#
10.1 When to Bring It Up#
- Any chat or collaboration design: presence, fan-out and ordering are the usual deep dive.
- A notification platform with an in-app inbox: cursors and sync are the same machinery.
- "A group of a million" or "three devices per user": the cue for read fan-out, passive sessions and per-device cursors.
10.2 Key Phrases#
FRAMING
"Three classes of event: durable messages, stored, ordered and retried; presence as soft state with a TTL; ephemeral events like typing that we're happy to drop."
PRESENCE
"Heartbeats with a TTL of d plus a bit, so a dead phone goes offline on its own, and fan-out only to people who have that user on screen. In my estimate that takes presence from twice the message traffic to half of it."
FAN-OUT
"Bodies once in the conversation log, pointers to small groups' inboxes, nothing per member in huge rooms, and live pushes only to sessions with the room open."
ORDERING AND DEDUP
"One owner per conversation assigns sequence numbers. Delivery is at-least-once, so sends dedup on the client message id and devices dedup on the sequence number."
10.3 Questions to Ask the Interviewer#
- How big can groups get, and do large ones need read receipts?
- Is history kept on the server, or only until delivery?
- How many devices per user?
10.4 Common Follow-Up Questions#
| Question | Strong answer |
|---|---|
| "How do you know a user went offline if their phone just dies?" | The heartbeat entry expires after d + ε; a delayed check at d + 2ε publishes offline. |
| "A celebrity with 2 million followers comes online. What happens?" | Little: only followers with them on screen are subscribed. |
| "Two people send at the same moment. Who's first?" | Whoever the conversation's owner sequences first; everyone sees that order. |
| "Can you guarantee exactly-once delivery?" | At-least-once with dedup on send and receive gives exactly-once display. |
| "Can sequence numbers repeat after failover?" | No: the new owner starts from the durable max, and a unique key rejects a stale owner. |
Chapter 11: Practice Problems#
Practice · problem 1
Problem 1: Presence for a 500,000-Member Community#
80,000 of 500,000 members are online during a match; product wants a member list with dots and "online now: 80,412". Whose presence does anyone actually see, and what happens when 60,000 people close the app at the final whistle?
Hint: visible-portion member lists, debounced downgrades, and a counter summed from per-shard counts every few seconds.
Practice · problem 2
Problem 2: Read Receipts for 1,024-Member Groups#
Design "message info" (who received and read each message) for groups of up to 1,024, computed from cursors.
Hint: count members whose read cursor is at or past the message's sequence number, on demand, when the sender opens the screen.
Practice · problem 3
Problem 3: Linking a New Laptop#
Priya links a laptop to two years of chats, and the server keeps messages only until delivery. Where does history come from, where do the laptop's cursors start, and how are messages sent during the transfer neither lost nor duplicated?
Hint: WhatsApp's approach: an encrypted history bundle from the phone, the laptop added to the device list for new messages, and app state synced separately (Meta Engineering, 2021).
Summary#
THREE CLASSES OF EVENT
- Durable (messages, read cursors): stored first, ordered, acked, retried
- Soft state (presence): in memory, TTL, latest wins
- Ephemeral (typing): throttled, expires on the receiver, dropped first
PRESENCE AND FAN-OUT
- Heartbeats with a TTL; per device underneath; last seen stamped by the server
- Visible-only subscriptions: ~84,000 down to ~22,000 pushes a second in our estimate
- Bodies once; pointers to small groups; nothing per member in huge rooms; one copy per node
ORDER, RECEIPTS, SYNC
- One owner per conversation assigns sequence numbers; a unique key fences failover
- Receipts are forward-only cursors; group ticks are the minimum
- At-least-once with dedup on both ends; push the hint, pull the truth
Further Reading#
Engineering Blogs#
- Blog
LinkedIn — Now You See Me, Now You Don't: LinkedIn's Real-Time Presence Platform (January 2018): https://www.linkedin.com/blog/engineering/product-design/now-you-see-me-now-you-dont-linkedins-real-time-presence-platf
- Blog
Slack — Real-time Messaging (April 2023): https://slack.engineering/real-time-messaging/
- Blog
Slack — Flannel: An Application-Level Edge Cache to Make Slack Scale (May 2017): https://slack.engineering/flannel-an-application-level-edge-cache-to-make-slack-scale/
- Blog
Discord — How Discord Scaled Elixir to 5,000,000 Concurrent Users (July 2017): https://discord.com/blog/how-discord-scaled-elixir-to-5-000-000-concurrent-users
- Blog
Discord — Maxjourney: Pushing Discord's Limits with a Million+ Online Users in a Single Server (October 2023): https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server
- Blog
Discord — Using Rust to Scale Elixir for 11 Million Concurrent Users (May 2019): https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users
- Blog
Discord — Why Discord Is Switching from Go to Rust (February 2020): https://discord.com/blog/why-discord-is-switching-from-go-to-rust
- Blog
Discord — How Discord Stores Billions of Messages (January 2017): https://discord.com/blog/how-discord-stores-billions-of-messages
- Blog
Meta — Building Mobile-First Infrastructure for Messenger (October 2014): https://engineering.fb.com/2014/10/09/production-engineering/building-mobile-first-infrastructure-for-messenger/
- Blog
Meta — How WhatsApp Enables Multi-Device Capability (July 2021): https://engineering.fb.com/2021/07/14/security/whatsapp-multi-device/
Official Documentation#
- Docs
Slack — User presence and status: https://docs.slack.dev/apis/web-api/user-presence-and-status
- Read
WhatsApp — Privacy Policy: https://www.whatsapp.com/legal/privacy-policy
- Docs
WhatsApp Help Center — How to check read receipts: https://faq.whatsapp.com/665923838265756
- Blog
WhatsApp — Communities now available (November 2022): https://blog.whatsapp.com/communities-now-available
this lesson in shorts · 4
play them all →One idea from this lesson in about a minute each, made for your phone.
Go deeper · reading trail
all reading →Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.
Now You See Me, Now You Don't: LinkedIn's Real-Time Presence Platform
Akhilesh Gupta, Meng Lay · 2018- What
- How LinkedIn shows the green online dot for its members in real time.
- Key idea
- Members heartbeat every d seconds, entries expire after d + ε, and one Akka actor per online member fires a delayed offline check at d + 2ε; changes go out on per-member topics.
- Why here
- This lesson's presence design almost line for line: a dead phone goes offline by itself, and short drops don't flap the dot.
Real-time Messaging
Sameera Thangudu · 2023- What
- An overview of the services that deliver Slack messages, presence and typing events to connected clients.
- Key idea
- Channel servers, mapped by consistent hashing, send each message once to every subscribed gateway server, which sends it to its clients; transient events like typing skip the database.
- Why here
- This lesson's fan-out: one copy per gateway node, presence from its own servers, and ephemeral events kept off the durable path.
Maxjourney: Pushing Discord's Limits with a Million+ Online Users in a Single Server
Yuliy Pisetsky · 2023- What
- How Discord made one server, Midjourney's, work with over a million users online at once.
- Key idea
- Fan-out work grows quadratically with server size, so Discord stopped sending passive sessions everything and split the work across relays of up to 15,000 sessions each.
- Why here
- This lesson's huge rooms: push full messages only to people looking, and split a hot room's fan-out across many workers.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.



