The Delivery Pipeline: Guarantees, Priorities and Preferences.

It's the super-app's festive sale, the Friday evening from Day 1's Preface. Marketing has a cashback push ready for 10 million users at 20:00. (An illustrative scene: the people, times and amounts are made up; each failure is a classic one.)

None of these is a provider bug. Each is a pipeline decision: which queue a message waits in, how often it can be sent, when preferences are read, whose clock decides, and what a retry does.

Day 1 drew the boxes: a Send API, an outbox, Kafka topics by priority, a router, a worker pool per channel. Today we build what runs through them, and the guarantee each stage makes.

01Chapter 1: The Pipeline in One Picture#

1.1 The Simple Definition#

A notification delivery pipeline turns a business event ("payment received", "new login") into zero or more messages on the user's channels, and keeps four promises:

1. Nothing accepted is lost. Once the API says yes, the notification survives crashes.
2. Nobody sees it twice. Retries happen everywhere; the user mustn't notice.
3. Urgent never waits behind bulk. An OTP doesn't queue behind a campaign.
4. The user's wishes win. Opt-outs, quiet hours and caps are applied as late as possible.

The pipeline ends when a provider (APNs, FCM, the email or SMS provider) accepts the message, or when the item is written to the user's in-app inbox. What happens on a live socket after that is Day 3.

1.2 The Stages#

%% title: The delivery pipeline: from an event to a provider
flowchart TD
  SVC@{ icon: "sd:server", label: "Payments, security,<br/>marketing services" } --> API["Send API<br/>validate, dedup on key"]:::core
  API --> PG@{ icon: "tech:postgresql", label: "Notifications + outbox<br/>one transaction" }
  PG --> REL["Outbox relay"]:::service
  REL --> LANES@{ icon: "tech:kafka", label: "Lane topics<br/>critical · transactional · bulk" }
  LANES --> RT["Router<br/>preferences, quiet hours,<br/>caps, batching"]:::core
  RT --> PREF@{ icon: "tech:redis", label: "Preference cache" }
  RT --> SCH["Scheduler and digests<br/>send later"]:::service
  SCH --> LANES
  RT --> IA["In-app worker<br/>inbox first"]:::service
  RT --> EW["Email worker"]:::service
  RT --> SW["SMS worker"]:::service
  IA --> IB@{ icon: "sd:database", label: "In-app inbox<br/>durable, seq per user" }
  IA --> PW["Push worker"]:::service
  PW --> APNS@{ icon: "tech:apple", label: "APNs" }
  PW --> FCM@{ icon: "tech:firebase", label: "FCM" }
  EW --> SG@{ icon: "tech:sendgrid", label: "Email provider" }
  SW --> SMS@{ icon: "sd:sms", label: "SMS provider" }
  PW & EW & SW -.->|"retries exhausted"| DLQ[["DLQ per channel"]]:::bad
  class SVC client
  class PG,IB store
  class PREF cache
  class LANES queue
  class APNS,FCM,SG,SMS external

Each stage adds one guarantee: ingest makes "accepted" durable, lanes keep bulk off critical, routing applies the user's wishes, workers deliver at-least-once, and tracking answers "did Priya get it?"

02Chapter 2: The Send API#

2.1 What a Request Carries#

{
  "idempotency_key": "payments:pay_8812:received",
  "user_id": "u_51ab",
  "category": "transaction",
  "template": "payment_received",
  "variables": {"amount_paise": 240000, "sender_name": "Arjun"},
  "priority": "transactional",
  "expires_at": "2026-10-30T20:12:00Z",
  "collapse_key": null,
  "send_at": null
}

• idempotency_key names the business event, not the HTTP request, so every retry from any pod carries the same key.
• expires_at says when the notification becomes worthless: an OTP in minutes, a bill reminder at the due date, a sale push when the sale ends. Every later stage checks it.
• collapse_key says "only the newest matters" (a badge, an ETA). It becomes apns-collapse-id and FCM's collapse key at the edge.

2.2 Idempotency Keys#

The caller will retry: its client timed out at 3 seconds, we answered at 3.1, and it can't tell "never arrived" from "answer lost". Apply Week 2, Day 2 · Idempotency in Practice: store the key with a unique constraint in the same transaction as the notification, and on a repeat return the original answer. Stripe's API is the public model: a retry with the same key returns the saved response, keys can be pruned "after they're at least 24 hours old", and reusing a key with different parameters is an error, "to prevent accidental misuse" (Stripe API docs).

# api/send.py: accept one notification (illustrative)
async def accept(req, db):
    if not valid(req):                                   # cheap checks only (2.3)
        return 400, {"error": problems(req)}
    async with db.transaction():
        row = await db.fetchrow(
            """INSERT INTO notifications (id, idem_key, req_hash, user_id, lane, body, status)
               VALUES ($1, $2, $3, $4, $5, $6, 'PENDING')
               ON CONFLICT (idem_key) DO NOTHING
               RETURNING id""",
            new_id(), req.idempotency_key, sha256(req.canonical()), req.user_id, lane_of(req), req.json())
        if row is None:                                  # seen before: same answer, no new send
            old = await db.fetchrow("SELECT id, req_hash FROM notifications WHERE idem_key = $1",
                                    req.idempotency_key)
            if old["req_hash"] != sha256(req.canonical()):
                return 422, {"error": "idempotency key reused with a different request"}
            return 202, {"notification_id": old["id"], "duplicate": True}
        await db.execute("INSERT INTO outbox (notification_id, topic) VALUES ($1, $2)",
                         row["id"], f"notifications.{lane_of(req)}")
    return 202, {"notification_id": row["id"]}

The outbox row is Week 3, Day 2 · Transactional Outbox: the notification and "please publish this" commit together, and a relay publishes to Kafka. The relay can crash between publishing and marking the row, so it may publish twice; everything downstream dedups on notification_id (Chapter 8).

2.3 Validation: Cheap and Synchronous#

The API rejects with 400 what can never succeed: an unknown template, missing variables, a past expires_at, a closed account. Queued, they'd only reach a DLQ.

It does not check preferences or quiet hours, which can change before the send. The API says "accepted"; the router decides "sent, held, batched or suppressed", and records which.

03Chapter 3: Preferences, Quiet Hours and Caps#

3.1 The Preference Model#

Preferences form a hierarchy, and the most specific setting wins: global ("pause everything"), category (marketing off, social on), channel within category (transactions: push and in-app, no email). Some categories are mandatory: security alerts, OTPs and regulatory notices can't be switched off, only given extra channels. A user with no preference row gets each category's defaults, and marketing is opt-in where the law demands it. Unsubscribes must bite fast: Gmail requires that "marketing messages and subscribed messages must support one-click unsubscribe" (Google sender guidelines).

3.2 Check Late, Then Check Again#

The router reads preferences from a cache invalidated on every change (Week 4, Day 2 · Cache Invalidation). But routing can happen long before the send: a campaign waits in the bulk lane, a reminder is scheduled weeks ahead.

So the channel worker re-checks, just before the provider call, what can change: the opt-out, quiet hours, expires_at, and any business condition ("is the bill still unpaid?"). A failed check marks the task suppressed or expired, with a reason. Uber's push pipeline does this: delivery runs "last-mile checks" and can mark a push "suppressed (if they failed some last-mile check) or simply expired (if it expired before getting a chance to be delivered)" (Uber Engineering, 2022).

3.3 Quiet Hours in the User's Clock#

Quiet hours live in the user's time zone, stored as an IANA zone (Asia/Kolkata, Australia/Sydney), never a fixed offset, because offsets move with daylight saving. US telemarketing rules use the same clock: no telephone solicitation "before the hour of 8 a.m. or after 9 p.m. (local time at the called party's location)" (47 CFR 64.1200(c)(1)). That rule covers marketing calls and messages to residential subscribers, with exemptions such as an established business relationship; it doesn't govern app push or email, so the principle carries over, not the law.

What a quiet window does depends on the lane, and inside the transactional lane on the class:

# routing/quiet_hours.py: when may this push go out? (illustrative)
from datetime import datetime, time, timedelta
from zoneinfo import ZoneInfo

def next_allowed(now_utc: datetime, tz: str, start=time(22, 0), end=time(8, 0)) -> datetime:
    local = now_utc.astimezone(ZoneInfo(tz))
    t = local.time()
    in_quiet = (t >= start or t < end) if start > end else (start <= t < end)
    if not in_quiet:
        return now_utc
    day = local.date() if t < end else local.date() + timedelta(days=1)
    wake = datetime.combine(day, end, tzinfo=ZoneInfo(tz))   # zoneinfo resolves DST for that date
    return wake.astimezone(ZoneInfo("UTC"))

The caller adds a random spread: a million held pushes should leave between 08:00 and 08:20, not in one second.

3.4 Frequency Caps#

Caps are counters with expiry per user and category, the limiters of Week 1, Day 3 · Rate Limiting at Scale keyed by user. Uber's example constraint is a "Daily frequency cap (e.g., at most 2 pushes per day)" (Uber Engineering, 2022). Cap the producers too: LinkedIn's gateway "will rate-limit upstream applications to prevent them from accidentally spamming members" (LinkedIn Engineering, 2018).

04Chapter 4: Templates and Localisation#

A template is keyed by (name, channel, locale, version): a push fits APNs' 4 KB payload (Apple docs), an SMS one segment if possible, an email needs a subject, HTML and plain text. Four rules:

1. Pin the version at accept time, so a retry hours later renders the same words.
2. Fall back through locales: es-MX → es → en-US. A missing translation mustn't fail a payment alert.
3. Format at render time in the user's locale and zone: amount_paise: 240000 becomes ₹2,400.00.
4. Keep secrets off lock screens: "New sign-in on Chrome, Windows"; details stay in the app.

Store a hash of what was rendered with the status, so Day 1's audit trail shows what each user saw.

05Chapter 5: Batching and Digests#

5.1 Collapse Before You Send#

Fifty pushes is a bug; "Arjun and 49 others reacted to your split" is a feature. Batching holds similar notifications briefly and sends one summary. LinkedIn's gateway "can delay specific notifications, place them in a notifications queue via RocksDB, and then send a group of notifications together" (LinkedIn Engineering, 2018). Flush rules: count (N items), window (T seconds after the first), or hybrid (whichever comes first).

Batching isn't collapsing. apns-collapse-id (at most 64 bytes) merges notifications on the device (Apple docs): it saves the screen but still costs every send. Batching saves the sends.

5.2 Batching Without Losing or Doubling#

The batch lives in Redis at batch:{user}:{category}, with a deadline set by its first item. Two races:

• Append during flush: RENAME the key when the flush starts, so new items open a new batch instead of vanishing.
• Two flushers: give each flush an id ({user}:{category}:{window_start}) and use it as the summary's idempotency key; the second copy is dropped downstream.

The items are already in the inbox (Chapter 7), so a lost batch costs one push, not the notifications.

5.3 Digests#

A digest is a batch on a schedule: "yesterday's transactions at 08:00 local". The scheduler (Chapter 10) fires one job per user per day into the bulk lane; the job skips the send if there's nothing to report, and uses digest:{user}:{date} as its key so a re-run sends nothing new.

06Chapter 6: Priority Lanes#

6.1 A Queue per Lane#

Put 10 million campaign messages in front of Rohan's OTP and it waits for 10 million sends: head-of-line blocking. Day 1's requirement was that transactions and security alerts "should never be delayed by marketing campaigns".

Each lane gets its own topic, consumer group, workers, retry topics and share of provider quota. Share any one and you've rebuilt the shared queue.

%% title: Lanes: separate queues, workers and provider quota
flowchart TD
  RT["Router"]:::core --> C[["notifications.critical<br/>OTP, fraud, login"]]:::hot
  RT --> T[["notifications.transactional<br/>payments, social, reminders"]]:::queue
  RT --> B[["notifications.bulk<br/>campaigns, digests"]]:::queue
  C --> CW["Critical workers<br/>always over-provisioned"]:::service
  T --> TW["Transactional workers"]:::service
  B --> TH["Campaign throttle<br/>token bucket"]:::service
  TH --> BW["Bulk workers"]:::service
  CW & TW & BW --> Q["Provider quota<br/>reserved per lane, bulk gets the rest"]:::core
  Q --> APNS@{ icon: "tech:apple", label: "APNs" }
  Q --> FCM@{ icon: "tech:firebase", label: "FCM" }
  Q --> SMSP@{ icon: "sd:sms", label: "SMS provider" }
  class APNS,FCM,SMSP external

6.2 Lanes Reach the Provider#

Providers have their own priorities, so the lane must reach their headers:

• APNs: apns-priority 10 sends "immediately"; 5 sends "based on power considerations", and 5 and 1 "might get grouped and delivered in bursts" (Apple docs). Critical and transactional: 10. Bulk: 5.
• FCM: high priority can "wake a sleeping device", but "if FCM detects a pattern in which messages don't result in user-facing notifications, your messages may be deprioritized to normal priority", judged over 7 days (Firebase docs). Silent marketing sent as high can cost your fraud alerts their urgency.
• Expiry: pass expires_at on as apns-expiration and FCM's time to live. Twilio's SMS validity period (default 36,000 seconds) bounds a message's wait in its queue; Twilio suggests 5–300 seconds for verification codes so "stale messages do not reach end users" (Twilio, 2021). Two clocks, then: our OTP code is valid for 5 minutes, and its SMS carries a 120-second validity period, the longest it may wait in the provider's queue before being dropped, so any code that does arrive still has three minutes to be typed. A late OTP is worse than a failed one.

Quota is shared across lanes: Firebase's HTTP v1 API "allots 600K Quota Tokens for each 1-minute Token Bucket" (Firebase docs). Reserve a slice for critical and transactional; bulk's token bucket spends what's left.

6.3 Ordering#

Within a lane, partition by user_id so one user's notifications stay in order. Across lanes there's no order, which is right: a fraud alert should overtake a reminder.

07Chapter 7: Fan-Out to Channels, Inbox First#

7.1 One Task per Channel#

The router turns one notification into channel tasks, keyed {notification_id}:{channel}, and per device for push (…:push:{device_id}; Day 1 assumed ~1.5 devices per user). Each channel has its own worker pool because each fails differently: APNs rejects dead tokens, email bounces hours later, SMS carriers throttle. A slow provider then backs up only its own workers (Week 3, Day 3 · Backpressure and Flow Control).

7.2 The Inbox Is Written Before Anyone Is Told#

Every user-visible notification that belongs in the app gets an in-app inbox row, and that write comes first:

1. The in-app worker appends the item to the user's inbox with the next per-user sequence number, durably.
2. Only then is anyone told: the live socket (Day 3), and a push if nobody is looking.
3. Email and SMS go out from their own workers, in parallel.

Nothing else is a record:

• The socket is at-most-once. Day 3's pub/sub drops what it carries during a gateway restart, which is fine only because the inbox has the item.
• Push stores almost nothing. APNs keeps one notification per app for an offline device; FCM keeps 100 on Android, then discards them all (Apple and Firebase docs).
• The app must match the doorbell. A "₹2,400 received" push that opens to an empty bell destroys trust in both.

The push is a doorbell; the inbox is the mailbox. Badges, "mark as read" and the escalation saga (Chapter 11) all read the inbox. Day 3 relies on this rule: its clients resume "after 1183" with a range read on this inbox, and its delivery service pushes only after the inbox write succeeds. OTPs are the exception: a code shouldn't sit in a list.

%% title: Inbox first: the durable write comes before any live delivery
sequenceDiagram
  autonumber
  participant R as Router
  participant W as In-app worker
  participant I as Inbox
  participant D as Live delivery (Day 3)
  participant P as Push worker
  participant E as Email worker
  R->>W: task n_77 in-app, for Priya
  W->>I: append n_77, next seq for Priya
  I-->>W: stored as seq 1184
  W->>D: hint seq 1184 to live sockets
  alt a live client acks within seconds
    D-->>W: acked, she has seen it
  else no socket or no ack, or a critical alert
    W->>P: push with collapse key, priority 10
  end
  R->>E: task n_77 email, in parallel if her preferences ask for it

7.3 Status Tracking#

%% title: Status of one channel task
stateDiagram-v2
  [*] --> PENDING
  PENDING --> QUEUED: outbox published
  QUEUED --> ROUTING: router picked it up
  ROUTING --> SUPPRESSED: preference, cap or quiet-hours drop
  ROUTING --> SENDING: channel task created
  SENDING --> SENT: provider accepted
  SENDING --> FAILED: retries exhausted, to DLQ
  SENDING --> EXPIRED: expires_at passed
  SENT --> DELIVERED: provider or device receipt
  SENT --> BOUNCED: bounce callback
  DELIVERED --> READ: opened in the app
  FAILED --> QUEUED: redrive
  class PENDING,QUEUED,ROUTING queue
  class SENDING core
  class SENT,DELIVERED,READ ok
  class FAILED,BOUNCED,EXPIRED bad

SENT means the provider accepted it, nothing more. DELIVERED needs a receipt: an email delivery webhook, an SMS delivery report, or the app reporting a push it received (FCM's HTTP v1 API has no per-message delivery callback; its delivery data comes as aggregates or a BigQuery export, after the fact). Webhooks are signed, so verify them, and they repeat, so a transition applies at most once and only moves forward.

08Chapter 8: At-Least-Once, Deduplicated#

8.1 Why Not Exactly Once#

A push worker gets 200 from APNs and crashes before committing its Kafka offset. Its replacement can't know the push went out. Whichever order you send and record in, a crash between them gives a duplicate or a loss. Kafka's documentation draws the line: "exactly-once delivery for other destination systems generally requires cooperation with such systems" (Apache Kafka docs). An SMS carrier won't cooperate, so choose per lane:

8.2 Dedup at Every Hop#

The worker's part is claim, send, record:

# workers/claim.py: at-least-once with a narrow duplicate window (illustrative)
async def deliver_once(task, provider, redis, lane_policy):
    key = f"sent:{task.key}"                             # n_77:push:dev_2
    if await redis.get(key) == b"done":
        return "duplicate-skipped"
    claimed = await redis.set(key, "sending", nx=True, ex=lane_policy.claim_ttl_s)
    if not claimed:
        state = await redis.get(key)
        if state == b"done" or lane_policy.at_most_once:  # bulk: unknown is good enough
            return "duplicate-skipped"
        return "retry-later"                            # someone is mid-send: don't race them
    result = await provider.send(task)                  # the only unrecoverable step
    if result.ok:
        await redis.set(key, "done", ex=7 * 86400)
    else:
        await redis.delete(key)                         # free the claim so a retry can send
    return result

If the worker dies between provider.send and the done write, the claim expires and a critical or transactional task is sent again: a duplicate by design, from a window of milliseconds. A bulk task treats the expired claim as "probably sent": at-most-once. The guarantee is a per-lane policy, not a property of the system.

09Chapter 9: Retries, Backoff and the DLQ#

9.1 Classify First#

Retrying a permanent failure wastes quota and delays everything behind it (Week 2, Day 4 · Webhook Delivery drew the line):

9.2 Backoff Bounded by Expiry#

Back off exponentially with full jitter, so a thousand failures don't return in the same instant. Firebase is specific: "Do not immediately retry a failed request with FCM. Wait at least 10 seconds before retrying" (Firebase docs). Then cut every schedule at expires_at:

# workers/retry.py: what to do with a failed send (illustrative)
import random

def next_step(task, result, now):
    policy = task.retry_policy                            # per lane; social and reminders back off further than payments
    if result.permanent:                                  # 410, 413, bad number...
        return ("fallback", task.fallback) if task.fallback else ("dlq", result.reason)
    if result.retry_after_s is not None:                  # the provider said when
        delay = result.retry_after_s
    else:
        cap = policy.base_delay_s * 2 ** task.attempt
        delay = max(policy.min_delay_s, random.uniform(0, cap))   # full jitter
    if task.attempt + 1 >= policy.max_attempts or now + delay >= task.expires_at:
        if task.lane == "bulk":
            return ("drop", "expired or out of attempts")
        return ("fallback", task.fallback) if task.fallback else ("dlq", "retries exhausted")
    return ("retry", delay)

9.3 Retry Topics, Not Sleeping Workers#

A worker sleeping 40 seconds holds its partition hostage. Publish the failed task to a delay topic for its lane (push.transactional.retry.40s), commit, move on: the delayed retry queue of Week 3, Day 4 · Dead Letters and Poison Pills. Final failures go to a DLQ per channel with the provider's reason and an alert on its growth; redrive re-enters the pipeline, so messages are deduped and re-checked for preferences and expiry. Provider-wide outages need circuit breakers and a second provider, not retries: that's Day 5.

10Chapter 10: Scheduling and Time-Zone Sends#

10.1 The Scheduler#

Reminders, digests and campaigns are rows with a send_at, kept in per-minute time buckets; workers owning shares of the buckets publish due rows to their lane (Week 2, Day 5 · Distributed Cron). Uber's push scheduler "has to provide a distributed cron-like system with a throughput upwards of tens of thousands triggers/second", and it "is idempotent, and allows a scheduled push to be rescheduled to another time" (Uber Engineering, 2022). A bucket loaded twice mustn't send twice, so the scheduled row's id is the idempotency key.

Cancel by condition, not by delete. The user pays a bill today; tomorrow's "bill due" reminder is already scheduled. Deleting it races the scheduler. Attach "bill still unpaid" instead, and let the last-mile check evaluate it.

10.2 "10:00 Local" for 50 Million People#

"10:00 in each user's zone" is one wave per zone. Store the campaign as (local time, zones) and expand each zone shortly before its 10:00, using that day's rules; UTC times computed weeks ahead break when daylight saving starts in between.

Then smear each wave. Firebase asks senders to "avoid sending messages within a 2 minute window of each of the :00, :15, :30, and :45 minute marks" and to ramp "from 0 to the max RPS across a 60 second time-window" at minimum (Firebase docs). Spread each wave over 10–20 minutes; nobody can tell 10:00 from 10:07. (LinkedIn and Uber go further and pick a send time per user, on the same kind of scheduler.)

11Chapter 11: The Escalation Saga#

11.1 Push, Wait, Then SMS#

The usual requirement is "if push fails, fall back to SMS". But an accepted push isn't a seen push. For a fraud alert the question is has she read it?, a process with timers:

1. Inbox row and push now.
2. Not read in 2 minutes: SMS.
3. Not read in 10 more: email.
4. Not read in 24 hours: flag the account for the fraud team.

This ladder is the rule for security and fraud alerts (Day 1's router policy). OTPs skip it: a code is useless late, so it goes by SMS at once.

This is a saga (Week 5, Day 2 · The Saga Pattern) with nothing to compensate: a sent SMS can't be unsent. Every step moves forward, and "undo" means cancelling the steps not yet taken once she reads the alert on any channel.

%% title: Escalation saga for a fraud alert
stateDiagram-v2
  [*] --> PUSHED: inbox write, push now
  PUSHED --> DONE: read within 2 min
  PUSHED --> SMS_SENT: no read after 2 min
  SMS_SENT --> DONE: read within 10 min
  SMS_SENT --> EMAILED: no read after 10 min
  EMAILED --> DONE: read
  EMAILED --> FLAGGED: no read in 24 h, fraud team told
  DONE --> [*]
  FLAGGED --> [*]
  class PUSHED,SMS_SENT,EMAILED core
  class DONE ok
  class FLAGGED bad

11.2 Durable Timers#

A sleep(120) in a worker dies with the next deploy, and the SMS never goes. Run escalation in a durable workflow engine (Week 5, Day 3 · Saga Orchestration), whose timers survive restarts, or as scheduler rows with a condition:

# escalation/fraud_alert.py: a durable workflow, Temporal-style (illustrative)
import asyncio
from datetime import timedelta
from temporalio import workflow

@workflow.defn
class FraudAlertEscalation:
    def __init__(self):
        self.read = False

    @workflow.signal
    def mark_read(self):                      # sent when the inbox item is read on any device
        self.read = True

    @workflow.run
    async def run(self, n):
        await workflow.execute_activity(send_channel, args=[n, "push"], **CRITICAL)
        for channel, wait in (("sms", 120), ("email", 600)):
            if await self._read_within(wait):
                return "read"
            await workflow.execute_activity(send_channel, args=[n, channel], **CRITICAL)
        if not await self._read_within(86_400):
            await workflow.execute_activity(flag_for_fraud_team, n.user_id)
        return "escalated"

    async def _read_within(self, seconds):
        try:
            await workflow.wait_condition(lambda: self.read, timeout=timedelta(seconds=seconds))
            return True
        except asyncio.TimeoutError:
            return False

Each send_channel uses the normal pipeline with its own key (n_91:sms), so a replayed step is deduped. Two races to name: a read at 2:00.4 against a timer at 2:00.0 sends one needless SMS (re-check read state at the last mile); and quiet hours, which a fraud alert ignores but a bill reminder's escalation respects, so the policy belongs to the category.

12Chapter 12: Queues and Workers for 20:00#

The preface's evening done properly: sign-in OTPs spike as the sale opens, and the 10M-user campaign launches at once. Two tools size it. Little's law: calls in flight = rate × time per call, which sets concurrency. Drain time: backlog ÷ the rate you're allowed to send, which is often the provider's quota, not your worker count.

So alert per lane: a few hundred messages of lag on notifications.critical pages someone; 10M on notifications.bulk at 20:00 is the plan, and alerts only if it's still draining at 20:40.

13Chapter 13: How Real Companies Do It#

LinkedIn: one gateway for every notification. Air Traffic Controller fronts LinkedIn's email, SMS, push and in-app notifications, "over a billion requests per day". It rate-limits senders, aggregates, picks channel and time, and dedups by "dropping any request it has already processed", with inputs partitioned "by member id". LinkedIn reports it cut member complaints in half, and P90 latency for message pushes fell "from about 12 seconds to about 1.5 seconds" (LinkedIn Engineering, 2018).

Uber: an inbox and a scheduler in front of push. Uber's Consumer Communication Gateway stores push content and metadata in an "inbox, which is a sharded array of MySQL datastores" (a store of pending pushes, not a user-facing inbox like ours), schedules pushes with an idempotent, reschedulable scheduler, and runs last-mile checks that can suppress or expire them (Uber Engineering, 2022).

Duolingo: four million pushes in seconds. For its 2024 Super Bowl ad, Duolingo reached about 4 million learners, "95% of eligible learners within 3.9 seconds and 99% within 5.7 seconds" (Duolingo, 2024). At QCon London 2024, Vitor Pellegrino and Zhen Zhou described the design: the audience prepared ahead in S3, tiers of SQS queues carrying users in batches ("500 iOS users in the same batch, 250 Android users"), a FIFO queue's 5-minute deduplication window so that several marketers pressing the button at once sent one notification, and a dedicated ECS cluster. That's the bulk lane at its extreme: precompute, batch, dedup the trigger, isolate the capacity.

14Chapter 14: Common Mistakes#

1. One queue for everything. A campaign blocks the OTPs. Separate topics, workers and quota.
2. Promising exactly-once. Say "at-least-once, dedup at every hop, collapse ids for the rare duplicate".
3. Checking preferences once. Check at routing and at the last mile, or unsubscribes and quiet hours lag behind the queue.
4. Scheduling in the server's clock. "20:00" for everyone is 01:30 in Sydney. Store IANA zones, expand per zone, smear each wave.
5. Retrying without classifying or expiring. Retrying a 410, or delivering an expired OTP, wastes quota and confuses users.

15Chapter 15: Interview Tips and Phrases#

15.1 When to Bring It Up#

Any notification prompt, and any design whose side effects leave your system. Reach for lanes when urgent and bulk traffic mix, and for the escalation saga when you hear "fall back to SMS if…".

15.3 Questions to Ask the Interviewer#

• Which notifications are mandatory, and which can users turn off?
• What's the latency goal per type, and how long is each worth sending?
• Is a rare duplicate acceptable for payment alerts?
• Do campaigns target local time across zones?

15.4 Common Follow-Up Questions#

16Chapter 16: Practice Problems#

Summary#

Further Reading#

Engineering Posts and Talks#

• Blog

  LinkedIn — Air Traffic Controller: Member-First Notifications at LinkedIn (1 March 2018): https://www.linkedin.com/blog/engineering/messaging-notifications/air-traffic-controller-member-first-notifications-at-linkedin

• Blog

  Uber — How Uber Optimizes the Timing of Push Notifications using ML and Linear Programming (3 November 2022): https://www.uber.com/blog/how-uber-optimizes-push-notifications-using-ml/

• Blog

  Duolingo — Super Bowl commercial 2024 (12 February 2024): https://blog.duolingo.com/super-bowl-commercial-2024

• Read

  Vitor Pellegrino and Zhen Zhou — How We Created a High-Scale Notification System at Duolingo (QCon London 2024): https://www.infoq.com/presentations/duolingo-high-scale-notification/

Documentation and Standards#

• Docs

  Apple — Sending notification requests to APNs · Handling notification responses from APNs: https://developer.apple.com/documentation/usernotifications/sending-notification-requests-to-apns · https://developer.apple.com/documentation/usernotifications/handling-notification-responses-from-apns

• Docs

  Firebase — Sending FCM messages at scale · Android message priority · Collapsible messages: https://firebase.google.com/docs/cloud-messaging/scale-fcm · https://firebase.google.com/docs/cloud-messaging/android/message-priority · https://firebase.google.com/docs/cloud-messaging/customize-messages/collapsible-message-types

• Blog

  Twilio — Validity Period for Promotional and Verification Use Cases: https://www.twilio.com/en-us/blog/developers/tutorials/product/validity-period-promotional-verification-use-cases

• Docs

  Stripe — Idempotent requests: https://docs.stripe.com/api/idempotent_requests

• Read

  Apache Kafka — Design: message delivery semantics: https://kafka.apache.org/40/design/design/

• Docs

  Google — Email sender guidelines: https://support.google.com/a/answer/81126

• Read

  47 CFR § 64.1200 (US telephone solicitation hours): https://www.law.cornell.edu/cfr/text/47/64.1200