~learn.
?

Day 0428 minPreface+60 XP

Presence and Chat Fan-Out at Scale.

5 stages4 code

watch it first · newThe whole lesson in a 24-minute animated lecture5 quiz checkpoints · ask the AI about any moment · captions, chapters and speed

Preface · 2 min

Preface#

Saturday night, the super-app's group chat "Goa Trip", six friends. Priya has paid for the villa and posts a split request. Yesterday's connections are all up. Everything on top of them goes wrong within half an hour. (An illustrative scene: the people, times and amounts are made up; each failure is a classic one.)

ONE EVENING, FIVE BUGS (ILLUSTRATIVE)
  • 23:31 —
    Priya sends "Paid ₹48,000 for the villa, ₹8,000 each please". The server stores it, but its acknowledgement is lost on lift Wi-Fi, so her app retries
  • 23:31 —
    Arjun sees the request twice, and pays twice
  • 23:33 —
    Rohan replies "done!". His phone clock is four minutes slow, so on Meera's laptop "done!" sorts above the request
  • 23:40 —
    Kabir's phone died at 23:12, and the gateway node holding his connection was restarted before it published "offline". Priya still sees him online. When he finally flips, his "last seen" says 23:58, taken from his phone's wrong clock
  • 23:44 —
    "Meera is typing…" has been on screen for nine minutes. The "stopped typing" event was dropped
  • 23:59 —
    The "Cricket Night" community, 300,000 members, 40,000 online, reaches the last over. Every message and presence change costs 40,000 deliveries; the chat tier's CPU hits 100%

None of these is a transport bug. What broke runs on top of the connections: who is online, who gets told, in what order, how many times, and what happens to people who weren't there.

ExplainerThe whole lesson: presence and chat fan-out

Day 4 runs on top of the sockets. An illustrative night in "Goa Trip" shows a lost ack and a double payment, a fast clock, a dead phone still online, a stuck typing indicator and a 300,000-member room at 100% CPU; events split into durable, soft state and ephemeral. Presence uses heartbeats with a TTL of d + ε and an offline trigger at d + 2ε, per device, with a server-stamped last seen, and visible-only subscriptions cut ~84,000 pushes a second to ~22,000 (assumed traffic). Messages are accepted by one owner per conversation (seq 412) and fanned out as bodies once with pointers to small groups, one copy per gateway node in hot rooms; receipts are forward-only cursors. Sync resumes from per-device cursors, sends and receives are deduped, and failures heal through TTLs and the log.

Yesterday (Day 3) built the pipes: gateways, a connection registry, one pub/sub channel per gateway, a per-user inbox that clients resume from, and the heartbeats, backpressure and reconnect handling that keep it all up. Today assumes they work and builds what runs through them.

Part I · Stage 1 of 5 · 8 min

Part I: Foundations#

Chapter 1: What Runs on Top of a Connection#

1.1 The Simple Definitions#

  • Presence: the system's best guess at whether a person is reachable now (online, idle, offline), plus last seen when they're not.
  • Fan-out: turning one event into one delivery per interested device.
  • Ordering: every member of a conversation sees its messages in the same order.
  • Receipts: how far a message got: stored, delivered to a device, read by a person.
  • Store-and-forward: what a device missed waits in an inbox until it reconnects and syncs.

1.2 Three Classes of Event#

These events are not the same kind of thing. Treat them alike and you get presence that's too expensive and messages that are too fragile.

Durable Soft state Ephemeral
Examples Messages, edits, membership, read markers Presence, last seen Typing, "recording audio"
If one is lost A bug: someone misses a payment request Heals at the next heartbeat Nobody notices
Stored? Yes, before anyone is told In memory, with a TTL No
Acked and retried? Yes, end to end No: the next update replaces it No
Ordered? Per conversation, by sequence number Latest wins, per user No
Under overload Queue it, never drop Coalesce to the newest state Drop first

Slack draws the same line: its transient events "are not persisted in the database and are sent through a slightly different flow", and typing is its example (Slack Engineering, 2023).

Chapter 2: Presence#

2.1 Heartbeats With a TTL#

Yesterday's heartbeat asked "is this socket alive?", and the connection registry answered "which gateway holds it?": routing state. Presence asks "is this person reachable?", and it has an audience. Each device tells the presence service it's still there every d seconds, and the service stores that with an expiry a little longer than d. If nothing renews it, the user goes offline. Nobody has to say goodbye, which matters because phones die, lose signal and get killed by the OS in silence.

LinkedIn published this design for its green dots (LinkedIn Engineering, 2018): a heartbeat every d seconds, an entry that expires after d + ε, and a delayed trigger after d + 2ε that publishes "offline" only if no newer heartbeat came. A member who "drops their connection and then reconnects within d seconds" stays online, which "prevents unnecessary fluctuations". That's the fix for Kabir: whatever crashed, he goes offline by himself.

DiagramPresence: heartbeats in, visible-only fan-out out
%% title: Presence: heartbeats in, visible-only fan-out out
flowchart TD
  P1@{ icon: "sd:phone", label: "Kabir's phone<br/>heartbeat every d s" }
  P2@{ icon: "sd:laptop", label: "Kabir's laptop<br/>lid closed, entry expired" }
  GW@{ icon: "sd:gateway", label: "Gateways<br/>from yesterday" }
  PS["Presence shard<br/>owns Kabir · hash of user id<br/>per-device entries, TTL d + ε"]:::core
  SUB@{ icon: "sd:cache", label: "Who is watching Kabir<br/>subscriptions from open screens" }
  W1@{ icon: "sd:phone", label: "Priya<br/>Goa Trip open" }
  W2@{ icon: "sd:phone", label: "Meera<br/>chat list visible" }
  X["Kabir's other 148 contacts<br/>not looking: told nothing"]:::service
  P1 --> GW
  P2 --> GW
  GW ==>|"heartbeat"| PS
  PS -->|"state changed?"| SUB
  SUB ==>|"one event each"| W1
  SUB ==>|"one event each"| W2
  SUB -.->|"skipped"| X
  class P1,P2,W1,W2 client
  class GW edge
  class SUB cache

2.2 Per Device or Per User?#

Store presence per device, and derive the user's state as the most present device: online if any device is active, idle if connected but inactive, offline when the last device expires. Slack's docs use this rule: a user is active "if they have at least one client connected to Slack, and they are not marked as 'away'", and "after 10 minutes with no activity, the user is automatically marked as away". Per-device state keeps a user online on the laptop when the phone dies, and tells delivery (Chapter 6) which devices are reachable.

Publish only when the derived user state changes. A laptop going idle while the phone is active changes nothing anyone sees, so nothing is sent.

2.3 Last Seen#

Last seen is the time of the user's last heartbeat. Stamp it with the presence server's clock when the heartbeat arrives, never the phone's (Chapter 7.2), and record the last heartbeat, not the moment the offline trigger fired. It's usually a privacy setting too, so it's checked against the viewer's permissions on read.

pythonPresenceShard · 35 lines
import time

ONLINE, IDLE, OFFLINE = 2, 1, 0
D, EPS = 30, 5            # heartbeat every 30 s, 5 s of slack (illustrative values)

class PresenceShard:
    """Presence for the users hashed to this shard. Memory only: it's soft state."""

    def __init__(self, publish):
        self.devices = {}     # user -> {device: (state, expires_at)}
        self.last_beat = {}   # user -> server wall-clock time of the last heartbeat
        self.shown = {}       # user -> state last published to watchers
        self.publish = publish

    def heartbeat(self, user, device, state=ONLINE):
        self.devices.setdefault(user, {})[device] = (state, time.monotonic() + D + EPS)
        self.last_beat[user] = time.time()          # the server's clock, never the phone's
        self._maybe_publish(user)

    def sweep(self):
        """The delayed trigger: runs every few seconds, drops devices silent for d + 2ε."""
        now = time.monotonic()
        for user, devs in list(self.devices.items()):
            for dev, (_, expires_at) in list(devs.items()):
                if now > expires_at + EPS:
                    del devs[dev]
            self._maybe_publish(user)

    def _maybe_publish(self, user):
        devs = self.devices.get(user, {})
        state = max((s for s, _ in devs.values()), default=OFFLINE)  # any device online wins
        if state != self.shown.get(user, OFFLINE):                   # publish changes only
            self.shown[user] = state
            last_seen = self.last_beat.get(user) if state == OFFLINE else None
            self.publish(user, state, last_seen)                     # to watchers only (2.4)

2.4 Who Gets Told#

Knowing presence is cheap; telling people is the expensive part. The naive design publishes every change to every contact. Estimate it before reading on.

Your turn · checkpoint 1not answered

Before the numbers: presence for 5 million chat users online

The super-app has 50 million users, and 7.5 million are online at the evening peak (Day 1). Assume 5 million of those are using chat, the only users whose presence is tracked and shown; the rest are paying bills or checking balances. Each chat user's state changes 4 times an hour after smoothing out flaps, and each has 150 contacts on the app, 10% of them online in chat. If every change goes to every online contact, how many pushes a second is that? Compare it with messages: the 10 million daily actives send 45 a day on average, peak is 3× average, and each message reaches 2.5 online devices.

Think it through, or jot it down. Then compare.

One strong answer

  • Changes: 5M × 4 ÷ 3,600 ≈ 5,600 a second.
  • Eager presence: 5,600 × 150 = 840,000 lookups a second; 10% online → ~84,000 pushes a second.
  • Messages: 450M a day ÷ 86,400 ≈ 5,200/s, ×3 ≈ 15,600/s, × 2.5 ≈ 39,000 deliveries a second.
  • So eager presence sends twice the product's message traffic, for a dot most recipients aren't looking at.

The fix is lazy, visible-only presence: a client subscribes to the users on its screen (the open chat, the visible part of the chat list) and unsubscribes as they scroll away. The published systems all do this:

  • Slack: "A Slack client receives presence notifications only for a subset of users that are visible in the app screen at any moment" (2023). Moving presence to subscriptions cut presence events reaching clients "by a factor of 5" (2017); the API has required subscriptions since January 2018 and calls "500 users ... a good maximum".
  • Discord: sends "the updates for the visible portion of the Member List" only, for "less network traffic, less CPU usage, better battery life" (2019).
  • LinkedIn: publishes each member's status on that member's own topic, which reaches the connections "who could be currently viewing the member's presence status" (2018).
WorksheetPresence at peak for the super-app (all inputs assumed except Day 1's users, daily actives and online share)
Value
Users / online at the evening peak (Day 1: 15%) / of those, in chat with presence tracked (assumed) 50 million / 7.5 million / 5 million
State changes per online user after debouncing (assumed) 4 an hour → ~5,600 a second
Eager: × 150 contacts, delivered to the 10% online in chat 840,000 lookups → ~84,000 pushes a second
Visible-only: online users watching each changing user (assumed) 4 → ~22,000 pushes a second
Visible-only without debouncing (30 raw flips an hour, assumed) ~42,000 changes → ~167,000 pushes a second
Message deliveries at peak (10M daily actives × 45 a day, ×3, × 2.5 devices) ~39,000 a second
Memory: 5M users × 1.3 devices × ~100 B, plus 100M subscriptions (~20 per online chat user, assumed) × ~32 B ~0.7 GB + ~3.2 GB

Conclusion: eager presence costs twice the message traffic; visible-only presence with debouncing costs about half of it, and all its state fits in memory on a few shards. Presence gets its own in-memory, user-sharded service and its own path to the gateways, and is never written to the message database.

Subscriptions are soft state too: after a shard restart, clients re-send their lists (Slack's presence_sub takes the "entire list of user IDs ... in each request") and heartbeats rebuild the states within d seconds.

2.5 Flapping and Debouncing#

A phone in a lift goes online, offline, online in ten seconds, and every flip fans out. Three dampers: the ε in d + 2ε absorbs short drops; delay downgrades, not upgrades (online shows at once, offline can wait a few seconds in case it reverses); and coalesce per watcher, replacing an unsent presence event for Kabir instead of queueing a second. Only the newest state matters, which is why presence can be dropped under backpressure. Slack's API has batched presence changes into one event, with a users array, since November 2017.

Chapter 3: Typing Indicators#

"Meera is typing…" is worth something for a few seconds and nothing after. A typical design (the values are illustrative):

  • The sender throttles: at most one typing event every 3 seconds while the user types.
  • The receiver expires: the indicator disappears 5 seconds after the last event, whether or not a "stopped" event arrives. That alone fixes Meera's stuck indicator.
  • The server only forwards: no storage, no sequence number, no ack, no retry, and only to members with the conversation open.
  • Overload drops it first, and big rooms show "3 people are typing" or nothing at all.
Your turn · checkpoint 2not answered

What breaks if typing goes through the message path?

A teammate suggests sending typing indicators through the message pipeline, so they get sequence numbers, durable storage and retries "for free". Each message takes about 10 seconds to type, so a throttled client sends ~3 typing events per message. What happens to storage, ordering and the user's experience?

Think it through, or jot it down. Then compare.

One strong answer

  • Volume: ~3 typing events per message roughly quadruples durable writes, for data that's worthless five seconds later.
  • Ordering: typing events take sequence numbers, so a gap-detecting client (Chapter 5) stalls or re-fetches a lost "typing" before showing the next real message.
  • Retries: a typing event retried 20 seconds late shows "Meera is typing…" after she's gone.
  • Sync: a phone back after a day replays thousands of stale typing events. Ephemeral events must be droppable; the message path is built never to drop.

Part II · Stage 2 of 5 · 7 min

Part II: Message Fan-Out#

Chapter 4: 1:1, Groups and Huge Rooms#

4.1 One Message, End to End#

Sending has two halves that fail differently. Accept: the device sends the message with its own client message id; the conversation's owner assigns the next sequence number, writes it to the durable log, and only then acks (one tick). Distribute: the message, or a pointer to it, goes to every recipient device, live if connected, through the inbox and sync if not; each device acks once it has stored it (delivered).

DiagramPriya's split request, from her phone to Arjun's
%% title: Priya's split request, from her phone to Arjun's
sequenceDiagram
  autonumber
  participant PP as Priya's phone
  participant CS as Chat service, owner of Goa Trip
  participant LOG as Conversation log
  participant IN as Members' inboxes
  participant GW as Arjun's gateway
  participant AP as Arjun's phone
  PP->>CS: send, client id c-77, "Paid 48,000..."
  CS->>LOG: append as seq 412, unique on conversation and seq
  LOG-->>CS: stored
  CS-->>PP: ack seq 412, one tick
  CS->>IN: pointer, Goa Trip up to 412, one per member
  CS->>GW: live copy for connected devices
  GW->>AP: message seq 412
  AP-->>CS: delivered up to 412
  CS-->>PP: delivered, two ticks once all members have it

Nothing is acknowledged before it's durable, and nothing is distributed before it's stored. The Discord side quest is designed the same way: writes are stored before anyone is told, so a dropped connection loses the live copy of a message, never the message.

4.2 Write Fan-Out vs Read Fan-Out#

It's the choice from Week 4, Day 4 · Feed Caching: do the work when the message is written, or when it's read?

  • Write fan-out (inbox per member): on send, append an entry to every member's inbox; a device syncs by reading one inbox from its cursor. Messenger's Iris works this way: "a totally ordered queue of messaging updates" per user "with separate pointers into the queue", each advanced when an update reaches its destination (Meta Engineering, 2014).
  • Read fan-out (log per conversation): write the message once; members read each conversation's log from their own cursor. Discord stores messages by channel, keyed ((channel_id, bucket), message_id), and loads a channel by querying its time buckets from the newest back until it has enough messages (Discord Engineering, 2017).
Where to do the fan-out work

Write fan-out · inbox per member

  • Send: N inbox writes
  • Sync: one inbox, one cursor
  • Unread counts: fall out of the inbox
  • Breaks at: big groups: 300,000 writes per message
  • Fits: 1:1 and small groups

Read fan-out · one log per conversation

  • Send: one append
  • Sync: one query per conversation the user is in
  • Unread counts: latest seq minus the member's read cursor
  • Breaks at: users in hundreds of quiet conversations, all asked "anything new?" on reconnect
  • Fits: large groups and channels

The usual answer is a hybrid: the body goes once into the conversation log, and small groups get a tiny pointer ("Goa Trip is now at 412") in each member's inbox, the same per-user inbox, with its per-user sequence, that yesterday's clients resume from. Sync reads the inbox to learn which conversations changed, then reads only those logs. Huge rooms get no per-member writes at all: members pull when they open the room, and badges compare cursors. Discord's Read States service, which keeps "track of which channels and messages you have read", is "accessed every time you connect to Discord, every time a message is sent and every time a message is read" (Discord Engineering, 2020).

DiagramHybrid fan-out: bodies once, pointers to small groups, nothing per member in huge rooms
%% title: Hybrid fan-out: bodies once, pointers to small groups, nothing per member in huge rooms
flowchart TD
  M["New message<br/>conversation, seq, body"]:::core
  LOG@{ icon: "sd:database", label: "Conversation log<br/>key: conversation id, seq" }
  Q{"Members?"}
  SMALL@{ icon: "sd:queue", label: "Member inboxes<br/>pointer: conversation, latest seq" }
  BIG["No per-member writes<br/>unread = latest seq − read cursor"]:::service
  LIVE["Live push<br/>only to sessions with the room open"]:::service
  GW@{ icon: "sd:gateway", label: "Gateways" }
  M ==> LOG
  LOG --> Q
  Q -->|"up to ~1,000, illustrative"| SMALL
  Q -->|"more"| BIG
  LOG --> LIVE --> GW
  SMALL -.->|"sync reads"| LOG
  class LOG store
  class SMALL queue
  class GW edge
Your turn · checkpoint 3not answered

Pick the fan-out for three conversations

(a) Priya and Arjun's 1:1 chat; (b) "Goa Trip", 6 members; (c) "Cricket Night", 300,000 members, 40,000 online, 5,000 with it open, 2 messages a second in the last over. For each, pick write fan-out, read fan-out or the hybrid, and say who gets a live push.

Think it through, or jot it down. Then compare.

One strong answer

  • (a) and (b): hybrid: the body once, a pointer into 2 or 6 inboxes. A reconnecting phone finds them in its own inbox. Live push to every connected device of every member.
  • (c): read fan-out. Pointers would be 300,000 × 2 = 600,000 inbox writes a second for one room. Store once; members pull when they open it.
  • (c) live: full messages only to the 5,000 open sessions. The other 35,000 get at most a batched unread bump.

4.3 Huge Groups and Hot Rooms#

Big rooms cost members × activity, and activity grows with members. Discord: "the amount of work needed to handle a discord server grows quadratically with the size of the server"; 1,000 people online each saying one thing is 1 million notifications, 100,000 is 10 billion (Discord Engineering, 2023). In 2017, with servers of "up to 30,000 concurrent users", publishing one event from a large server took "900ms to 2.1s" at peak.

A hot room is a hot key (Week 1, Day 4 · Hot Keys and Skew), and the chat versions of the fixes are:

  1. One copy per server, not per session. Yesterday's one channel per gateway already gives this for the live path. Discord's Manifold sends to the remote nodes, so the sender calls send no more often than "the number of involved remote nodes" (2017). Slack's channel servers send each message to every subscribed gateway server, and "each GS that receives that message sends it to every connected client subscribed to that channel id" (2023).
  2. Push only to people looking. Discord stopped sending passive sessions everything from a server until the user clicks in; about 90% of connections in large servers were passive, making fan-out "90% less expensive" (2023).
  3. Split the room's work. Discord's relays do "fanout with permission checks" for up to 15,000 sessions each, so a server with "over 10 million members, of whom over 1 million are online" keeps working (2023).
  4. Thin the events: no typing or member presence in big rooms, aggregated reactions, slow mode.
  5. Isolate it, so a cricket final doesn't slow Priya's 1:1 chat on the same shard.
DiagramCricket Night: one copy per gateway node, full messages only to people looking
%% title: Cricket Night: one copy per gateway node, full messages only to people looking
flowchart TD
  OWN["Room owner<br/>Cricket Night · seq 88,104"]:::hot
  R1["Relay 1<br/>~15,000 sessions"]:::service
  R2["Relay 2<br/>~15,000 sessions"]:::service
  R3["Relay 3<br/>~10,000 sessions"]:::service
  G1@{ icon: "sd:server", label: "Gateway node A<br/>one copy in" }
  G2@{ icon: "sd:server", label: "Gateway node B<br/>one copy in" }
  ACT@{ icon: "sd:users", label: "Active sessions<br/>room open · full message" }
  PAS@{ icon: "sd:users", label: "Passive sessions<br/>badge bump, batched" }
  OWN ==> R1
  OWN ==> R2
  OWN ==> R3
  R1 ==> G1
  R1 ==> G2
  G1 ==> ACT
  G2 -.-> PAS
  class G1,G2 edge
  class ACT,PAS client

Chapter 5: Ordering and Receipts#

5.1 Per-Conversation Sequence Numbers#

Rohan's reply jumped the queue because clients sorted by the sender's clock. Yesterday's inbox numbered events per user, which orders one person's notifications. Chat needs a number every member agrees on, so give every conversation one owner that assigns sequence numbers:

  • Partition by conversation_id (Week 1, Day 1 · Partitioning), so one owner hands out 1, 2, 3… and every device shows that order.
  • Gaps are visible: a device holding 410 that receives 412 fetches 411.
  • Order is per conversation only: on the spectrum in Week 5, Day 1 · Consistency Models that's far weaker than one global order, and it's exactly what a log partitioned by conversation gives you (Week 3, Day 1 · Queue vs Stream). A total order would need one global sequencer for no visible benefit.

Ids can still be time-sortable for storage (Discord's Snowflakes are "chronologically sortable", 2017); what users see is fixed by one owner, not by senders' clocks. When the owner fails over, the new one starts from the durable max, and a unique key on (conversation_id, seq) makes a slow old owner's write fail: the key is the fence, as with fencing tokens in Week 5, Day 5 · Leader Election.

pythonConversationOwner · 17 lines
class ConversationOwner:
    """Owns some conversations (by hash of conversation_id). One writer per conversation."""

    def __init__(self, log):
        self.log = log          # durable; unique key (conversation_id, seq); also stores client ids
        self.next_seq = {}      # in memory, rebuilt from the log after a failover

    def append(self, conv_id, sender_device, client_msg_id, body):
        prior = self.log.find_client_id(conv_id, sender_device, client_msg_id)
        if prior is not None:                    # a retried send: same message, same seq, same ack
            return prior
        if conv_id not in self.next_seq:         # new owner: trust the log, not memory
            self.next_seq[conv_id] = self.log.max_seq(conv_id) + 1
        seq = self.next_seq[conv_id]
        self.log.insert(conv_id, seq, sender_device, client_msg_id, body)  # raises if seq is taken
        self.next_seq[conv_id] = seq + 1
        return seq                               # ack to the sender only after the insert

Each device keeps the highest sequence it has shown contiguously, drops duplicates, holds early arrivals briefly and fetches holes:

pythonConversationView · 16 lines
class ConversationView:
    def __init__(self, shown_upto=0):
        self.shown_upto = shown_upto   # every seq <= this is on screen
        self.early = {}                # seq -> message that arrived ahead of a gap

    def receive(self, msg, fetch_range):
        if msg.seq <= self.shown_upto or msg.seq in self.early:
            return []                  # duplicate from at-least-once delivery: drop it
        self.early[msg.seq] = msg
        shown = []
        while self.shown_upto + 1 in self.early:
            self.shown_upto += 1
            shown.append(self.early.pop(self.shown_upto))
        if self.early:                 # a hole: ask for it rather than wait for a retry
            fetch_range(self.shown_upto + 1, min(self.early) - 1)
        return shown

5.2 Delivery vs Read Receipts#

Receipt Means Reported by
Sent (one tick) Stored durably The owner, after the log write
Delivered (two ticks) A recipient device has stored it That device, after persisting it
Read (blue) A person has seen it The app, when the conversation is on screen

WhatsApp's help centre (2022 version) adds the group rule: the second tick appears when every participant has received the message, and the blue ticks when every participant has read it; read receipts are always sent in groups, even if you've turned them off.

Two choices keep receipts cheap:

  • Receipts are cursors, not per-message rows. "Delivered up to 412" and "read up to 410": one small event for any number of messages, one row per member per conversation.
  • Cursors only move forward. Apply max(old, new), so late, duplicated or reordered receipts can't move a tick backwards.

A group's tick is then the minimum cursor over members. WhatsApp caps groups at 1,024 (WhatsApp, 2022), which keeps that and the "message info" screen cheap; in a 300,000-member room, per-member receipts are meaningless. Read state gets hot: Discord saw "tens of thousands of database writes per second" and schedules "a database commit for 30 seconds in the future whenever a Read State is updated" (2020), so repeated updates to one cursor become one write.

DiagramA message's status as the sender sees it: it only ever moves forward
%% title: A message's status as the sender sees it: it only ever moves forward
stateDiagram-v2
  [*] --> Sending: tapped send
  Sending --> Sending: no ack, retry same client id
  Sending --> Sent: owner stored it
  Sent --> Delivered: every member has it on a device
  Delivered --> Read: every member read cursor past it
  Sent --> Read: read cursor arrives first
  Read --> [*]

Part III · Stage 3 of 5 · 5 min

Part III: Offline, Multi-Device and Reliability#

Chapter 6: Store-and-Forward and Sync#

6.1 The Inbox and Sync on Reconnect#

Most recipients aren't connected when a message is sent. WhatsApp's privacy policy: "If a message cannot be delivered immediately (for example, if the recipient is offline), we keep it in encrypted form on our servers for up to 30 days as we try to deliver it", and "Once your messages are delivered, they are deleted from our servers." Discord and Messenger keep history permanently and Slack does on paid plans (its free plan hides messages after 90 days), but catching up works the same way: a cursor per device, and a sync that sends everything after it.

Yesterday's resume was a range query on the per-user inbox; for chat, that query returns pointers, and the logs supply the bodies. The device sends its cursors, the server streams what changed, oldest first, and each ack advances the cursor and drives the senders' "delivered" ticks. After a long absence, send the newest page of each conversation first and load older history on scroll. Messenger calls this "a push-based snapshot + delta model", one snapshot then pushed deltas, and its new sync protocol cut non-media data use by "about 40%" (Meta Engineering, 2014).

DiagramKabir's phone comes back after 40 minutes offline
%% title: Kabir's phone comes back after 40 minutes offline
sequenceDiagram
  autonumber
  participant KP as Kabir's phone
  participant SY as Sync service
  participant IN as Kabir's inbox
  participant LOG as Conversation logs
  KP->>SY: resume, inbox cursor 9,120
  SY->>IN: entries after 9,120
  IN-->>SY: Goa Trip up to 415, Mom up to 88
  SY->>LOG: Goa Trip after 409, Mom after 86
  LOG-->>SY: 6 messages and 2 messages
  SY-->>KP: batch, oldest first
  KP->>KP: store, drop any already held
  KP-->>SY: ack, inbox cursor 9,122
  SY-->>KP: live from here
pythonsync · 11 lines
def sync(device, inbox, logs, page=200):
    """Everything this device missed, oldest first, a bounded page per conversation."""
    changes = inbox.read_after(device.inbox_cursor)          # pointers: (conv_id, latest_seq)
    batch = []
    for conv_id, latest in changes:
        have = device.cursors.get(conv_id, 0)
        if latest - have > page:                             # far behind: newest page now,
            batch.append(("older_available", conv_id, have)) # the rest loads on scroll
            have = latest - page
        batch.extend(logs.read(conv_id, after=have, upto=latest))
    return batch   # the device stores it, then acks: cursors advance, senders get "delivered"

6.2 Multi-Device Sync#

  • Cursors are per device. The laptop can be at 415 while a tablet off for a week is at 302.
  • The sender's other devices are recipients. WhatsApp's client encrypts each message "N number of times to N number of different devices — those in the sender and receiver's device lists" (Meta Engineering, 2021).
  • Read state syncs across devices. Reading on the laptop clears the phone's badge and pending notification: a read cursor is a tiny durable event, fanned out to the user's other devices.
  • A new device needs history. WhatsApp's phone sends a new device "a bundle of the messages from recent chats", app state is synced through encrypted copies on the server, and a phone can link "up to four other nonphone devices" (2021).

6.3 At-Least-Once Delivery With Client Dedup#

Every hop retries: sends, pushes, sync batches. Exactly-once delivery isn't available over an unreliable network; at-least-once plus idempotent processing is (Week 2, Day 2 · Idempotency in Practice, and Day 2 of this week for the notification pipeline). Chat needs two dedup points: the owner dedups sends on the client message id before assigning a sequence number (find_client_id in 5.1), and devices dedup receives on the sequence number (ConversationView).

Your turn · checkpoint 4not answered

The ack that never arrived

The owner stores Priya's request as seq 412, but the ack is lost and her phone retries after 5 seconds. Separately, Arjun's phone stores 412 from the live push, its "delivered" ack is lost, and sync re-sends 412. With no dedup anywhere, what does everyone see? Which two checks fix it, keyed on what?

Think it through, or jot it down. Then compare.

One strong answer

  • Without dedup: the retry becomes seq 413, so everyone sees the request twice (Arjun pays twice), and Arjun's phone may show 412 twice too.
  • On send: dedup on (sender device, client message id) before a seq is assigned, remembered for as long as a phone might retry (days). The retry gets "412" back.
  • On receive: dedup on (conversation, seq): 412 is already held, so it's dropped and re-acked.
  • Not timestamps or text: two identical "ok" messages a second apart are both real.

Chapter 7: Reliability Edge Cases#

7.1 Split-Brain Presence#

A partition leaves two presence nodes each believing they own Kabir. Half his friends see him online, half offline. Don't fix this with consensus on every heartbeat, for a green dot. Instead:

  • Let it heal. Presence is soft state with a TTL; once ownership settles, the next heartbeat is the truth and the wrong node's entry expires.
  • Version changes per user. The owner stamps each change with an increasing version (ownership epoch in the high bits); watchers keep only the highest, so a stale "offline" can't overwrite a newer "online".
  • Never use presence for correctness. Never skip storing a message or a push because presence said "online". A wrong dot is cosmetic; a dropped message isn't.

7.2 Clock Skew in Last Seen#

Kabir's "last seen 23:58" came from his phone's clock. Stamp last seen on the server; keep it monotonic per user (max(stored, new)), so moving between presence servers with slightly different clocks can't make it go backwards; show it coarsely ("today at 23:12") and never in the future; and never order messages by it, or by any phone timestamp.

7.3 Partial Fan-Out Failures#

The owner stores seq 412, then pushes to 40 gateway nodes and 6 inboxes. Node 17 times out, one inbox write fails, or the owner crashes halfway. At scale, this happens daily.

  • The log write is the commit point. Once 412 is in the log, the message exists.
  • Retry per recipient, not per message; receive-side dedup absorbs any repeats.
  • Drive fan-out from the log. A worker consumes the log from a checkpoint, so a crashed owner's unfinished fan-out resumes where it stopped: the idea behind Week 3, Day 2 · Transactional Outbox.
  • Sync is the safety net. A device that missed 412 has its cursor at 411; the next sync, or the gap when 413 arrives, fetches it.
  • Read membership as of the message. If Rohan leaves at seq 413, a slow fan-out of 412 still reaches him and 414 doesn't.
Your turn · checkpoint 5not answered

Partition, skew and a half-finished fan-out

The presence tier splits for 90 seconds during a deploy; Meera's laptop and phone are on opposite sides. Meanwhile the Goa Trip owner stores seq 420 and crashes after pushing it to 3 of the 6 members. What does each friend see, during and after, and which mechanism repairs each piece?

Think it through, or jot it down. Then compare.

One strong answer

  • Presence: some friends may see Meera offline during the split. Afterwards, heartbeats re-establish her within d, and per-user versions stop a stale "offline" winning. Nothing durable depended on it.
  • Message 420: it's in the log, so it exists. The fan-out worker resumes from its checkpoint for the other 3; failing that, their devices fetch 420 on the next sync or when 421 reveals the gap.
  • Ticks: Priya's two ticks appear once all 6 delivered cursors pass 420. Cursors only move forward, so retries can't confuse them.
FAILURE MATRIX
Failure Users see Contained by
Device dies silently "Online" for d + 2ε Heartbeat TTL
Presence shard restarts Blank dots for seconds Clients re-send subscriptions
Presence split-brain Mixed dots for one heartbeat TTL, per-user versions
Send ack lost Spinner, then a retry Dedup on client message id
Live push lost Message a little later Gap check, sync from the cursor
Owner crashes mid-fan-out Some members get it late Log commit, fan-out checkpoint
Owner fails over Short pause in one chat Durable max, unique (conversation, seq)
Hot room Slow delivery in that room Per-node copies, passive sessions, relays

Part IV · Stage 4 of 5 · 2 min

Part IV: Real-World Application#

Chapter 8: How Real Companies Do It#

Company What it has published
LinkedIn (2018) Heartbeats every d s, entries expiring after d + ε, one Akka actor per online member as a delayed trigger at d + 2ε; presence on per-member topics, reaching connections who may be viewing it; millions online, a status change reaches a viewer in under 200 ms at p99
Slack (2017, 2023, API docs) In-memory presence servers with users hashed to them; presence only for users visible on screen; subscriptions cut presence events 5×; channel servers by consistent hashing send one copy per gateway server; transient typing events skip the database; delivery "across the world in 500ms"
Discord (2017–2023) One process per server routing everything; quadratic fan-out; Manifold one copy per node; visible-portion member lists; passive sessions (90% of large-server connections); relays of 15,000 sessions; channel-keyed storage with Snowflake ids; Read States with 30-second commits
WhatsApp (policy, help centre, 2021, 2022) Undelivered messages kept encrypted up to 30 days, delivered ones deleted; group ticks only when all participants have received or read; groups up to 1,024; per-device encryption, history bundles and up to 4 linked devices
Messenger (2014) Iris: a totally ordered per-user queue with a pointer per destination; snapshot then pushed deltas; ~40% less non-media data

Chapter 9: Common Mistakes to Avoid#

9.1 Mistake 1: Presence on the Message Path#

Presence storage

WRONG: presence as durable data

  • Every change written, replicated and retried like a message
  • A reconnect storm becomes a database write storm
  • A presence surge delays real messages

CORRECT: presence as soft state

  • In memory, sharded by user, with TTLs
  • Its own path; coalesced or dropped under load
  • Only last seen is persisted, lazily

9.2 Mistake 2: Telling Everyone#

Presence fan-out

WRONG: broadcast to all contacts

  • ~84,000 pushes a second in our estimate, twice the messages
  • Most recipients aren't looking

CORRECT: visible-only subscriptions

  • Subscribe to who is on screen; debounce and coalesce
  • No member presence at all in huge rooms

9.3 Mistake 3: Ordering by Timestamp#

Message order

WRONG: sort by the sender's clock

  • A slow clock puts a reply above its question
  • Devices can disagree, and gaps are invisible

CORRECT: per-conversation sequence numbers

  • One owner assigns 1, 2, 3…; every device agrees
  • Gaps are detected and fetched; a unique key fences failover

9.4 Mistake 4: Treating the Live Push as Delivery#

What "delivered" means

WRONG: pushed to the socket = done

  • A half-failed fan-out goes unnoticed
  • A device reconnecting at that moment never gets it

CORRECT: log, then hint, then cursor

  • The log write commits; devices ack what they've stored
  • Sync and gap checks repair every missed push

Part V · Stage 5 of 5 · 4 min

Part V: Interview Preparation#

Chapter 10: Interview Tips and Phrases#

10.1 When to Bring It Up#

  • Any chat or collaboration design: presence, fan-out and ordering are the usual deep dive.
  • A notification platform with an in-app inbox: cursors and sync are the same machinery.
  • "A group of a million" or "three devices per user": the cue for read fan-out, passive sessions and per-device cursors.
❯ Say this

10.2 Key Phrases#

Key phrases
FRAMING

"Three classes of event: durable messages, stored, ordered and retried; presence as soft state with a TTL; ephemeral events like typing that we're happy to drop."

PRESENCE

"Heartbeats with a TTL of d plus a bit, so a dead phone goes offline on its own, and fan-out only to people who have that user on screen. In my estimate that takes presence from twice the message traffic to half of it."

FAN-OUT

"Bodies once in the conversation log, pointers to small groups' inboxes, nothing per member in huge rooms, and live pushes only to sessions with the room open."

ORDERING AND DEDUP

"One owner per conversation assigns sequence numbers. Delivery is at-least-once, so sends dedup on the client message id and devices dedup on the sequence number."

10.3 Questions to Ask the Interviewer#

  • How big can groups get, and do large ones need read receipts?
  • Is history kept on the server, or only until delivery?
  • How many devices per user?

10.4 Common Follow-Up Questions#

Question Strong answer
"How do you know a user went offline if their phone just dies?" The heartbeat entry expires after d + ε; a delayed check at d + 2ε publishes offline.
"A celebrity with 2 million followers comes online. What happens?" Little: only followers with them on screen are subscribed.
"Two people send at the same moment. Who's first?" Whoever the conversation's owner sequences first; everyone sees that order.
"Can you guarantee exactly-once delivery?" At-least-once with dedup on send and receive gives exactly-once display.
"Can sequence numbers repeat after failover?" No: the new owner starts from the durable max, and a unique key rejects a stale owner.

Chapter 11: Practice Problems#

Practice · problem 1

Problem 1: Presence for a 500,000-Member Community#

80,000 of 500,000 members are online during a match; product wants a member list with dots and "online now: 80,412". Whose presence does anyone actually see, and what happens when 60,000 people close the app at the final whistle?

Hint: visible-portion member lists, debounced downgrades, and a counter summed from per-shard counts every few seconds.

Practice · problem 2

Problem 2: Read Receipts for 1,024-Member Groups#

Design "message info" (who received and read each message) for groups of up to 1,024, computed from cursors.

Hint: count members whose read cursor is at or past the message's sequence number, on demand, when the sender opens the screen.

Practice · problem 3

Problem 3: Linking a New Laptop#

Priya links a laptop to two years of chats, and the server keeps messages only until delivery. Where does history come from, where do the laptop's cursors start, and how are messages sent during the transfer neither lost nor duplicated?

Hint: WhatsApp's approach: an encrypted history bundle from the phone, the laptop added to the device list for new messages, and app state synced separately (Meta Engineering, 2021).

Summary#

DAY 4 KEY TAKEAWAYS
THREE CLASSES OF EVENT
  • Durable (messages, read cursors): stored first, ordered, acked, retried
  • Soft state (presence): in memory, TTL, latest wins
  • Ephemeral (typing): throttled, expires on the receiver, dropped first
PRESENCE AND FAN-OUT
  • Heartbeats with a TTL; per device underneath; last seen stamped by the server
  • Visible-only subscriptions: ~84,000 down to ~22,000 pushes a second in our estimate
  • Bodies once; pointers to small groups; nothing per member in huge rooms; one copy per node
ORDER, RECEIPTS, SYNC
  • One owner per conversation assigns sequence numbers; a unique key fences failover
  • Receipts are forward-only cursors; group ticks are the minimum
  • At-least-once with dedup on both ends; push the hint, pull the truth

Further Reading#

Engineering Blogs#

Official Documentation#

this lesson in shorts · 4

play them all →

One idea from this lesson in about a minute each, made for your phone.

End of Day 04 · Presence and Chat Fan-Out at Scale
Finish Day 4Stages clear as you read them (0/5 so far). Skipping ahead doesn't count: marking complete unlocks after ~5 more min of reading.

Go deeper · reading trail

all reading →

Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.

LinkedInBlog post · ~10 min

Now You See Me, Now You Don't: LinkedIn's Real-Time Presence Platform

Akhilesh Gupta, Meng Lay · 2018
What
How LinkedIn shows the green online dot for its members in real time.
Key idea
Members heartbeat every d seconds, entries expire after d + ε, and one Akka actor per online member fires a delayed offline check at d + 2ε; changes go out on per-member topics.
Why here
This lesson's presence design almost line for line: a dead phone goes offline by itself, and short drops don't flap the dot.
taught here: Presence as soft state
SlackBlog post · ~10 min

Real-time Messaging

Sameera Thangudu · 2023
What
An overview of the services that deliver Slack messages, presence and typing events to connected clients.
Key idea
Channel servers, mapped by consistent hashing, send each message once to every subscribed gateway server, which sends it to its clients; transient events like typing skip the database.
Why here
This lesson's fan-out: one copy per gateway node, presence from its own servers, and ephemeral events kept off the durable path.
taught here: Fan-out through gateways
DiscordBlog post · ~15 min

Maxjourney: Pushing Discord's Limits with a Million+ Online Users in a Single Server

Yuliy Pisetsky · 2023
What
How Discord made one server, Midjourney's, work with over a million users online at once.
Key idea
Fan-out work grows quadratically with server size, so Discord stopped sending passive sessions everything and split the work across relays of up to 15,000 sessions each.
Why here
This lesson's huge rooms: push full messages only to people looking, and split a hot room's fan-out across many workers.
taught here: Hot rooms

New lessons by email

New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.

Discussion · …

Public: everyone reading this lesson sees comments. Be kind and keep it on the material.

Sign in to ask a question or answer one. Reading stays open to everyone.
loading the discussion…