Part 1System Design Fundamentals

HLD vs LLD. High-level design is the what: components, how they interact and why. Low-level design is the how: schemas, API contracts, algorithms, error handling. Interviews start high-level, then deep-dive one or two areas.

The four-step framework. Clarify requirements (3–5 min), estimate scale (5–7), draw the high-level design (15–20), then deep dive (10–15): schema, API, critical algorithms, edge cases and failures.

Functional vs non-functional. Functional requirements say what the system does; non-functional ones say how well: uptime, latency, throughput, consistency. The non-functional ones drive architecture, technology and cost.

Sync vs async. Sync is simple with immediate feedback, but the client blocks and services couple. Async through a queue decouples, absorbs spikes and allows retries, at the cost of eventual consistency and harder debugging.

API styles. REST for CRUD and public APIs, GraphQL when clients ask for exactly the data they need, gRPC (binary Protocol Buffers) for fast internal calls, WebSocket for server push.

• Designing before clarifying scope, scale and the read/write ratio.
• Skipping non-functional requirements: they decide the architecture.
• Diving into low-level detail before the high-level picture is agreed.

Part 2Infrastructure Building Blocks

CDN. Edge servers (PoPs) serve content near users: ~5–10 ms from a Mumbai PoP versus ~150–200 ms to a US origin. Cache static assets and identical-for-everyone responses; never user-specific data.

Versioned URLs. Put a content hash in the filename (app.a1b2c3d4.js) and cache it for a year as immutable. A deploy changes the URL, so users get fresh files with zero purge delay.

no-cache doesn't mean "don't cache". no-cache caches but revalidates (ETag, 304 Not Modified); no-store never caches; private keeps it out of the CDN; s-maxage gives the CDN its own TTL.

L4 vs L7 load balancing. L4 sees IP and port: very fast (<100 μs added), any protocol. L7 sees the whole HTTP request: routes by path, host, header or cookie and terminates TLS, adding 1–5 ms.

Shallow vs deep health checks. Point the load balancer at a shallow liveness check. A deep check that queries the database marks every server unhealthy when the DB is down; use deep checks for monitoring.

Stateless servers scale. Keep no session in server memory: sessions in Redis, data in PostgreSQL, files in S3. Then any server takes any request and a crash loses nothing.

• Caching user-specific or authenticated responses at the CDN.
• A deep health check behind the load balancer: one database outage removes every server.
• Microservices for a small team with unclear requirements; start with a monolith.

Part 3Back-of-the-Envelope Estimation

Per day to per second. A day is 86,400 seconds; round to 100,000. So X per day ÷ 100,000 ≈ X per second: 1 billion a day is about 10,000 a second.

The four estimates. Traffic (RPS, read/write split, peak), storage (objects × size × retention × replication), bandwidth (RPS × payload, ingress and egress) and infrastructure (servers, replicas, cache).

Latency ratios matter. L1 cache 0.5 ns, memory 100 ns, SSD random read 150 μs, HDD seek 10 ms; datacenter round trip 0.5 ms, cross-continent 150 ms. Cache aggressively, batch, and minimize round trips.

Media dominates storage. For a Twitter-like service, text is 30 GB a day but images and videos add ~7 TB a day: media is 99%+ of storage. Egress for images is why the CDN is essential.

Formulas to say out loud. RPS = daily actions ÷ 100,000. API servers = peak RPS ÷ 5,000 ÷ 0.7. Read replicas = read RPS ÷ 30,000. Cache = hot data: 20% of the total, or the last N days.

• Forgetting growth: 100 GB now can be 1 TB in a year at 10x growth.
• Forgetting replication (500 GB × 3 = 1.5 TB) or that images make posts 100x bigger.
• Mixing megabytes and megabits: MB/s × 8 = Mbps.

Part 4Networking Fundamentals for System Design

TCP guarantees, at a price. TCP gives reliable, ordered delivery with flow and congestion control, but the handshake costs 1 RTT before the first request and one lost packet blocks everything behind it.

UDP when late data is useless. No handshake, an 8-byte header, message boundaries kept, no reliability. Use it for real-time video, voice, gaming, DNS and as QUIC's base.

HTTP/1.1 → 2 → 3. HTTP/1.1 sends one request at a time per connection. HTTP/2 multiplexes streams and compresses headers over one TCP connection. HTTP/3 runs QUIC over UDP: a lost packet only blocks its own stream.

Reuse connections. Each new connection pays TCP and TLS handshakes and starts in slow start. Keep-alive and connection pools avoid that; short connections also pile up in TIME_WAIT and exhaust ephemeral ports.

Scaling WebSockets. Connections are stateful, so the load balancer needs stickiness, and users on different servers reach each other through a pub/sub backbone (Redis, Kafka). Use SSE for one-way server push.

• Opening a new connection per request instead of pooling: handshakes, slow start, port exhaustion.
• WebSocket for request-response or one-way push, where HTTP or SSE is simpler.
• Assuming HTTP/2 solved head-of-line blocking: it still rides one TCP connection.

Part 5Operating System Concepts for System Design

Processes isolate, threads share. A process owns its memory and file descriptors, so one crash kills only itself. Threads share the heap and descriptors with their own stacks: cheap to create and switch, but one crash kills them all.

Event loops scale connections. Thread-per-connection doesn't scale past ~10K connections. epoll registers descriptors once and returns only the ready ones, so one thread serves thousands (Node.js, nginx, Redis).

Context switches cost more than they look. Saving registers is cheap; the TLB flush, cold CPU caches and pipeline flush are not. Use async I/O, thread pools, batching and CPU affinity to switch less.

Write-ahead logging. Append the change to a sequential log and fsync it before touching data files; on a crash, replay the log. Sequential HDD writes run ~100 MB/s versus ~1 MB/s random.

Containers are namespaces plus cgroups. Namespaces isolate what a process sees (PID, network, mounts, users); cgroups limit what it uses (CPU quota, memory). Exceed the memory limit and the container is OOM killed.

• Thread-per-connection for a million connections instead of an event loop.
• Taking locks in different orders across threads: a deadlock waiting to happen.
• Running a database that caches pages itself without O_DIRECT, so data is cached twice.

Part 6System Design Terminology & Theory

CAP is C or A during a partition. Partitions will happen, so P isn't optional. During one, a CP system refuses what it can't confirm (ZooKeeper, etcd); an AP system answers with possibly stale data (Cassandra, DNS).

PACELC. Even with no partition, strong consistency costs latency: synchronous replication waits for the slowest replica. DynamoDB and Cassandra are PA/EL; HBase is PC/EC.

Isolation levels. Read uncommitted allows dirty reads; read committed (PostgreSQL's default) allows non-repeatable reads; repeatable read (MySQL's default) allows phantoms; serializable allows none.

Consistency models. Linearizability respects real-time order; sequential keeps only each process's program order; causal keeps cause before effect (vector clocks); eventual only promises convergence.

Delivery semantics. At-most-once may lose messages; at-least-once may duplicate them, so consumers must be idempotent; exactly-once needs idempotent producers and consumers, or transactional messaging.

Error budgets. A 99.9% SLO allows 43.2 minutes of downtime in a 30-day month. Budget left: ship fast. Budget gone: freeze features and fix reliability. Keep the SLA looser than the SLO.

• "Pick 2 of 3" in CAP: partition tolerance is a given, not a choice.
• Assuming exactly-once delivery is free instead of making consumers idempotent.
• Saga compensations that aren't idempotent: they may run more than once.

Part 7API Design for System Design

Resources, not verbs. Model things as plural nouns with a few standard methods: Get, List, Create, Update, Delete. A real verb stays rare and attached to a resource: POST /v1/orders/{id}:cancel.

Paginate with cursors from day one. Adding pagination later breaks clients. Keyset on (sort key, unique ID) with an opaque token costs the same on every page; offset walks every skipped row and duplicates or skips rows under writes.

Additive changes only. Within a version, add endpoints, optional request fields and response fields. Renames, removals, type changes, new required fields and changed defaults break clients. Clients ignore unknown fields.

Idempotency keys on POST. The client sends one key per logical operation and reuses it on every retry. A repeat returns the original response; the same key with a different body is an error.

Errors are a contract. Clients branch on a stable code; people read the message. RFC 9457 problem details, plus the field and whether to retry. One shape for every error, including the gateway's 429s.

Slow work becomes a resource. Return 202 with a status resource to poll, whose terminal states never change. Servers also get a webhook, mobile apps a push. Big results are a link to object storage.

• POSTs without an idempotency key: every timeout on an order or payment can become a duplicate.
• Renaming or retyping a field inside a version instead of adding alongside.
• Slow work inside the request instead of 202 and an operation.

Part 8Databases in Depth

A commit is one sequential write. Every change is appended to the WAL first; a commit is durable once its log record is flushed, while changed pages are written later. Group commit lets one fsync cover many transactions.

Composite index order. Equality columns first, then the one range or sort column. (user_id, created_at, id) reads 20 adjacent entries for a user's latest orders; INCLUDE the payload to make it covering.

Lost update and write skew. Read-then-write code loses updates on both PostgreSQL's and InnoDB's defaults: use a guarded atomic update or FOR UPDATE. Write skew needs Serializable or a constraint like UNIQUE (customer_id, coupon_code).

Where MVCC keeps old versions. PostgreSQL writes a new tuple in the heap and VACUUM removes the old one. InnoDB updates in place and keeps old versions in undo, which purge discards. A long transaction blocks both.

Optimistic concurrency. Each row has a version; the update runs WHERE version = seen. Zero rows means someone changed it: reload and decide again. It spans a user's think time, and maps to ETag and 412.

Time-ordered IDs. UUIDv4 inserts at random index leaves, so the whole index becomes the working set. UUIDv7 and Snowflake IDs sort by time and keep inserts at the index's right edge.

• Trusting the default isolation level with read-then-write code: neither prevents lost updates.
• An index for every slow query. Each taxes every write; read EXPLAIN first.
• Long transactions on the primary: they block vacuum and purge. Run reports on a replica.

Day 1Partitioning Deep-Dive

Partitioning is how you scale writes. Vertical scaling hits a ceiling and read replicas don't fix write bottlenecks. Partitioning (sharding) splits data across machines, each owning a subset, so storage, throughput and memory add up.

Hash partitioning. hash(key) % N spreads keys evenly with an O(1) lookup, but range queries become scatter-gather across every partition, and changing N moves most keys.

Range partitioning. Contiguous key ranges keep a range query on one partition, but auto-increment IDs or timestamps send every new write to the last partition.

Directory partitioning. A lookup table maps keys to partitions: any key can go anywhere and rebalancing is a directory update, but the directory is a single point of failure and adds a lookup to every request.

Consistent hashing. Keys and nodes sit on a ring; a key belongs to the first node clockwise. Adding a node moves only about 1/(N+1) of keys, from one neighbour. Virtual nodes even out the load.

• Partitioning by hash(user_id + timestamp): perfect spread, but no more queries by user_id alone.
• Range partitions on auto-increment IDs or time, so every new write lands on one partition.
• Over-partitioning early: 1000 partitions × 100 app servers = 100,000 connections.

Day 2Replication Trade-offs

Three reasons to replicate. Replicas buy high availability (fail over to a copy), read scaling (reads spread across copies; writes don't) and geographic latency (read from a nearby copy).

Sync, async, semi-sync. Sync waits for followers: no loss, slower writes. Async acks at once: fast, but a dead leader loses unreplicated writes. Semi-sync waits for one follower: the production middle ground.

Quorums. With N replicas, if W + R > N every read overlaps the latest write. N=3 with W=2, R=2 is strongly consistent; W=1, R=1 is faster but only eventual.

Replication lag anomalies. Lagging followers break read-your-writes (your update vanishes), monotonic reads (data goes back in time) and consistent prefix (a reply shows before its message).

Split brain needs fencing. A network partition can leave two leaders accepting writes. Fence the old one: STONITH, storage or network fencing, or a lease it must keep renewing in ZooKeeper or etcd.

• Serving a balance, inventory count or permission check from an async replica.
• Failing over without fencing the old primary: two primaries, conflicting data.
• Last-write-wins across leaders: clocks disagree and writes silently vanish.

Day 3Rate Limiting at Scale

Limits must be counted together. With 50 servers each seeing a slice of a user's traffic, uncoordinated local counters let 100 requests through a 10-per-minute limit. Counts must be shared, synced or sticky.

Fixed window's boundary burst. One counter per window is simple and O(1), but a user can spend a full quota just before and just after the boundary: 2x the limit within seconds.

Sliding window counter. Weight the previous window by its overlap and add the current one: (8 × 75%) + 3 = 9. Two counters per key, no boundary burst; the default for API limits.

Token bucket allows controlled bursts. Tokens refill at a fixed rate up to a capacity and each request spends one: the long-term rate is bounded, bursts up to capacity pass. Compute refill from elapsed time.

Fail open or fail closed. When the limit store dies, over-allow user-facing and paid APIs; under-allow DDoS protection, cost control and login. A conservative local fallback sits in between.

• Each server counting alone: users spread requests and multiply the limit.
• One blanket fail-open rule: an AI call costing $0.01 each should fail closed.
• Leaky bucket for user-facing limits: it queues requests and adds latency even with spare capacity.

Day 4Hot Keys and Skew

Traffic follows Zipf's law. The nth most popular key gets about 1/n of the top key's traffic. With 1,000 keys the top 1% take ~40%; viral content pushes the top 0.01% past 90%.

Detect with a Count-Min Sketch. Fixed memory, O(depth) per update: hash the key into several counters and read the minimum. It may overestimate but never underestimates, which is all hot key detection needs.

Layer the defenses for hot reads. CDN edge, then local app memory, then Redis with split keys and request coalescing; the database sees only misses. In the lesson, traffic past the edge drops from 10,000 to ~2,800 req/s.

Write-heavy hot keys can't be cached. Buffer increments and flush periodically, shard the counter across N keys (reads sum all N), or use a G-Counter CRDT where each node owns its own count.

Bifurcate by popularity. Celebrities get their own code path: posts from accounts under 10K followers are pushed into feeds; celebrity posts are pulled at read time and merged. Show approximate counts like "1.2M likes".

• Assuming uniform traffic: one viral key turns a partition into a single-node bottleneck.
• Letting a hot key expire all at once: thousands of requests stampede the database.
• Splitting keys that are written often: every write fans out to all splits.

Day 5Session Store Design

Sessions are read-heavy and forgiving. Every request validates a session (80% reads), so latency must stay under 5 ms P99. Durability is best effort: a lost session means logging in again, inconvenient but not catastrophic.

Sticky vs distributed sessions. Sticky sessions keep state in one server's memory: fastest, but a server failure logs its users out and deploys lose sessions. A distributed store (Redis) costs a network hop and survives both.

Client-side sessions can't be revoked. A JWT or encrypted cookie needs no storage and scales anywhere, but you can't invalidate it before expiry, it's capped at 4 KB and rides on every request.

Region-affine sessions. Encode the home region in the session ID (sess_east_abc123) and route there. If a region dies, its users re-authenticate; make that painless with SSO or remember-me.

Session security basics. 256-bit random IDs, a fresh ID on login to block fixation, and cookies set HttpOnly, Secure and SameSite=Lax.

• Local session caches without short TTLs or pub/sub invalidation: logged-out sessions keep working.
• A user_sessions set without the sessions' TTL: it fills with expired IDs.
• Sequential, guessable session IDs instead of secure random ones.

BossDesign a URL Shortener at Scale

Clarify, then estimate. Ask before designing: spikes, code alphabet, duplicate URLs, analytics freshness, API rate limits. The answers set the target: 120,000 redirects/s during viral spikes.

Pre-allocated key ranges. A key service hands each API server a range of 10,000 counter values, scrambled and base-62 encoded to 7 characters. Only range allocation needs coordination; a crash just loses a few thousand codes.

Split redirect from creation. Redirects are 10x writes and need 99.99% uptime, creation 99.9%. Separate services scale and fail independently, and clicks go to Kafka asynchronously so counting never slows a redirect.

Hybrid sharding. URLs live on shards chosen by consistent hashing on short_code (fast redirects); a light user_urls index sharded by user_id serves "my URLs". Every create writes both.

• Auto-increment codes (predictable), random-then-check (races) or UUIDs (36 characters, not 7).
• A bare 301 redirect: browsers cache it forever, hiding clicks and ignoring deletes.
• Caching URLs without their expiry, so expired links keep redirecting.

Day 1Timeout Hell

Slow is worse than down. A down service fails fast; a slow one holds your threads. With 100 threads, a 10 s downstream cuts you from 1,000 to 10 requests a second. Timeouts turn slow into down.

Percentiles, not averages. Set a timeout at the ~65 ms average and 5% of healthy requests fail. Rule of thumb: timeout = P99 × 2–3, so P99 = 200 ms gives 400–600 ms.

Timeout budgets. Give each dependency a share of the SLA, not the whole SLA. For a 5 s payment: 4.5 s budget, fraud check 600 ms, bank 3.5 s, notification (optional) 300 ms: 4.4 s in all.

Deadline propagation. Pass the absolute deadline down the chain, so each hop waits only for the time left. Per-hop 5 s timeouts across three services can add up to 15 s after the user gave up at 5.

Connection vs read timeout. Connection timeout covers the TCP handshake and should be short (1–5 s): if you can't connect, something is badly wrong. Read timeout covers the work and depends on the operation.

• No timeout, or 30 s "to be safe": threads pile up and you look dead.
• Timeout = average or P99, failing healthy requests every day.
• Adaptive timeouts without alerting on the timeout value and a hard maximum.

Day 2Idempotency in Practice

Request lost or response lost? A client can't tell them apart: both look like a timeout. No timeout tuning fixes that; the only answer is making the operation safe to retry.

The idempotency key. The client sends a unique key per logical operation. The server records it with status "processing" before calling the bank, stores the result, and returns the cached result for any repeat.

Pass the key downstream. Send your idempotency key to the bank too: it deduplicates as well, and on an ambiguous timeout you can ask it what happened to that key, then update your record.

Guard the edge cases. Hash the request body so a reused key with a different amount is rejected. Only completed requests block retries; a failed one may run again. Extend the TTL on completion.

• POST payments with no idempotency key: a double-click charges twice.
• A TTL shorter than the retry window, so a late retry processes again.
• Marking a partial failure "failed": a retry charges the customer twice.

Day 3Circuit Breakers

Fail fast when it's clearly broken. Timeouts and retries still wait 3.5 s per call and pile load onto a failing bank. A circuit breaker stops calling: users get an error in ~10 ms and the bank can recover.

Closed, open, half-open. Closed: calls pass and failures are counted. Open: calls fail immediately until a timer expires (often 30–60 s). Half-open: one test call decides, closed on success or open again.

Count what's really a failure. Timeouts, 5xx and refused connections count. 4xx errors and business rejections like "card declined" don't: the service worked correctly and said no.

Use failure rate, not count. "5 failures in 60 s" trips on Black Friday at 10,000 req/s even though 0.05% is normal. Open at a failure rate (around 50%) with a minimum number of calls.

Retry, breaker and bulkhead together. Check the circuit breaker first (cheapest), take a bulkhead slot second, retry transient failures last, with exponential backoff and jitter. They're layers, not alternatives.

• Closing to full traffic at once: the backed-up herd crashes the recovering service. Ramp up.
• Alerting only on error rate: an open circuit lowers it and hides the outage.
• A fallback for payments, auth or inventory: you can't fake those.

Day 4Webhook Delivery System

At-least-once plus idempotent receivers. True exactly-once is impossible (the Two Generals problem). Retry until acknowledged and give every event an ID the receiver uses as its idempotency key; a duplicate costs one lookup.

The pipeline. Event sources publish to a durable bus (Kafka); a dispatcher fans each event out to its subscriptions; delivery queues partitioned by subscription feed workers; exhausted retries go to a dead letter queue.

Retry what can succeed. Retry network errors, timeouts, 5xx and 429 (honour Retry-After). A 4xx means the request is malformed: retrying won't help, so log it or dead-letter it.

Sign and timestamp every webhook. HMAC-SHA256 over the exact bytes sent, with a timestamp inside the signature. Receivers compare in constant time and reject anything more than 5 minutes old, which stops replays.

Recover fast after an outage. Monitor endpoint health. When a dead endpoint recovers, move its pending retries to an immediate queue and ramp up gradually, within its rate limit, instead of waiting out the backoff.

• Re-serializing the payload after signing, so the receiver's HMAC over the raw body fails.
• Retrying 4xx responses that will never succeed.
• Delivering events for one entity in parallel: partition by entity to keep order.

Day 5Distributed Cron & Job Scheduling

Locks alone don't work. A plain lock stays held when its server crashes. A lock timeout lets a slow job run twice. A Redis lock released with GET then DELETE can delete the next holder's lock.

Perfect is impossible, so be practical. FLP says consensus can't be guaranteed with even one faulty process. Production cron runs at-least-once, makes jobs idempotent, tolerates small delays and alerts on missed runs.

One leader schedules. Elect a leader with a lease in etcd, ZooKeeper or Consul; followers wait to take over. Use a coordination service instead of building your own unless you build databases.

Fencing tokens stop stale leaders. Each new leader gets a higher token and sends it with every operation; resources reject older tokens. A deposed leader paused by GC can't do damage when it wakes up.

Track jobs outside the leader. Record executions in a job store with heartbeats. A new leader skips jobs still beating, a stuck-job detector reschedules silent ones, and checkpoints let a job resume instead of redoing work.

• A lock with no expiry: one crash and the job never runs again.
• Releasing a lock you may no longer own; check and delete atomically.
• Non-idempotent jobs: a failover rerun sends the same 50 emails again.

BossThe Ultimate System Design Interview

The spike is the problem. 5 million users hit one 10-minute sale and 80% of traffic lands in the first 30 seconds: ~150,000 requests a second at peak. CDN, WAF and rate limits absorb the herd before it reaches services.

Atomic claims in Redis. A Lua script checks the user's existing claim and decrements the inventory counter in one atomic step, restoring it if it goes below zero. No database in the hot path, so no overselling.

Claims expire back into the pool. A claim holds a unit for 3 minutes. A claim-expiry worker returns unpaid units to inventory, triggering a second wave of claims (~15,000 claim operations for 10,000 units).

Checkout uses every Week 2 pattern. Validate the claim, check the idempotency store by claim_id, call Stripe behind a circuit breaker with a timeout and its idempotency key, create the order in a transaction, publish events asynchronously.

Precompute notifications. Build the 2.4M-recipient list 10 minutes before the sale into Redis batches. At start, 24 workers fan out at once and queue everything in ~10 seconds; push and email have separate pools.

• Checking stock then decrementing in two steps: a race that oversells.
• Computing millions of notification recipients at the moment the sale starts.
• Putting the database in the claim path at 40,000 claim attempts a second.

Day 1Queue vs Stream

Work to do vs things that happened. A queue holds jobs: each message goes to one worker and is deleted after the ack. A stream is an append-only log: events stay, each reader tracks its own offset and can replay.

Consumer groups. Kafka's model: every group gets all messages (fan-out) and within a group each partition goes to exactly one consumer (load balancing). Parallelism equals the partition count.

Order lives in the partition. Kafka orders messages only within a partition. Key by the entity that needs order (order_id, user_id) so "created, paid, shipped" can't be processed out of order.

At-least-once plus idempotent consumers. Exactly-once needs special support on both sides, so most systems ack after processing and dedupe, e.g. by partition and offset stored in the same transaction as the work.

• Kafka for simple background jobs that a queue with built-in retry would handle.
• No partition key, so one order's events land on different partitions.
• One partition "to start": only one consumer can ever work on it.

Day 2Transactional Outbox

The dual-write problem. Saving an order then publishing to Kafka are two writes with no shared transaction. A crash in between leaves an order no downstream service hears about; flipping them publishes events for orders that never saved.

Write the event to the database. Insert the business row and an outbox row in the same transaction: both commit or neither does. A separate publisher reads unpublished outbox rows and sends them to Kafka.

Polling or CDC. A poller queries published_at IS NULL every few seconds: simple, but adds latency and database load. CDC (Debezium) tails the write-ahead log: ~10–100 ms latency, no polling, more to operate.

Still at-least-once. A publisher that crashes after Kafka acks but before marking the row published sends it again. Consumers dedupe by event ID; Kafka transactions can't cover the database update.

• Writing the outbox row in a separate transaction: the dual write is back.
• Publishing to Kafka inside the request path, so Kafka trouble fails user requests.
• No alert on outbox lag: the table grows to 10M rows before anyone notices.

Day 3Backpressure and Flow Control

Async removes natural backpressure. A synchronous client can't send faster than the server answers. A queue absorbs the gap instead, until memory runs out: absorption, then resource exhaustion, then collapse.

Watch lag, depth and latency. Consumer lag (alert at 10,000 messages or 5 minutes), queue depth (100,000 or growing for 5 minutes) and queue wait time. Throughput falling while lag grows means consumers are struggling.

Four responses. Shed low-priority load (metrics, logs), rate-limit producers (adaptively, as lag grows), scale consumers (Kafka: up to the partition count), or spill overflow to disk or S3 for later.

You get two of three. Completeness, latency, stability: pick two. Most systems keep completeness and stability and accept higher latency during spikes; real-time pricing drops stale events instead.

Recover gradually. After an outage, don't jump from 100 to 10,000 messages a second: the backlog floods downstream. Step up (500, 1,000, 2,500, 5,000, 10,000) after a minute of stability each.

• Unbounded in-memory queues: the OOM killer takes every message with it.
• Rate limiting only at the API gateway while internal services overwhelm the database.
• Hot partition keys: one consumer drowns while the others idle.

Day 4Dead Letters and Poison Pills

A poison pill blocks everything behind it. A message that crashes the consumer every time causes a crash loop: restart, fetch the same message, crash. In the lesson, one bad message held 50,000 orders for 4 hours.

Classify the failure. Transient (timeouts, rate limits): retry with backoff. Permanent (bad JSON, missing field, rule violation): straight to the DLQ. Poison pills (crashes, hangs, memory blowups): quarantine immediately.

Delayed retry topics. Instead of retrying in place, republish to retry topics with growing delays (1 s, 10 s, 60 s), then the DLQ. The main topic keeps flowing; Uber described this design for Kafka.

A DLQ is an operational tool. Store the original message, error and stack trace, retry count, timestamps and source topic, partition and offset. Monitor depth, growth and age, triage daily, and build replay tooling from day one.

Replay needs idempotency. Check the payment_id before reprocessing so a replay can't double-charge. Old messages may not match today's schema: version them, migrate on replay, or close them out.

• Resetting the retry count to 0 in each consumer instead of reading it from headers: infinite loop.
• A DLQ nobody monitors: it becomes a graveyard of lost data.
• A DLQ consumer that processes instead of just storing and alerting.

Day 5Audit Log System

Who did what, to which resource, when, from where. An audit log serves auditors, legal and security, not engineers: strict schema, retention in years, and it must be immutable. An application log is for debugging and can be deleted.

Capture in the same transaction. Write the audit event to an outbox in the same transaction as the business action, so a transfer can't succeed without its record. Kafka, consumers and the DLQ carry it asynchronously.

Immutable and tamper-evident. No UPDATE or DELETE: corrections are new events. Each event carries the previous event's hash, checkpointed somewhere an attacker can't change, and cold files sit under S3 Object Lock.

Tiered storage. Hot (0–90 days) in TimescaleDB for fast investigation queries, warm (to 1 year) in ClickHouse or BigQuery, cold (1–7 years) as Parquet on S3, queryable with Athena.

Never shed audit events. When consumers fall behind, scale them, batch harder and alert, but accept latency: audit data can't be dropped. Sort investigations by ingestion time and keep both timestamps.

• Writing the audit event after the business transaction: money moves with no trail.
• "Fixing" audit rows with UPDATE, which breaks the chain of custody.
• Logging only "delete doc_123" without actor, context or reason.

BossEvent-Driven Food Delivery Order System

Size the dinner rush. 500,000 orders a day, 60% in 6–9 PM: ~28 orders/s, ~56 at the peak, design for 100. With ~8 events per order and 7 consumer services, that's 5,600 event deliveries a second.

Kafka keyed by order_id. Seven services need the same events, replay helps debugging, and each order's events must stay in order. Partition orders.events (32 partitions) by order_id; each service is its own consumer group.

Different consumers, different guarantees. Payment needs effectively-once with an idempotency key per order (Stripe won't charge twice). Analytics can lag, push is fire-and-forget, fraud detection needs real time.

Retry topics, then the DLQ. Transient failures go to retry topics delayed 1, 5 and 15 minutes; after 3 retries, the DLQ. Payment failures retry then notify the customer; restaurant rejections find an alternative or refund.

Audit for disputes. 2% of orders are disputed. Every service writes immutable audit events, so support queries one order's full timeline in minutes instead of correlating logs across services.

• Saving the order and publishing to Kafka as two writes: lost or phantom orders.
• Shedding driver-assignment events: a late driver hurts more than late analytics.
• Retrying forever in the main topic, so one failing order blocks the rest.

Day 1Caching Patterns

Why cache. Latency (database 10–100 ms, Redis 0.5–2 ms, local memory 0.0001 ms), throughput (5,000 database queries/s vs 100,000+ Redis ops/s) and cost. "Just add Redis" isn't a strategy.

Cache-aside is the default. The app checks the cache, on a miss reads the database and fills the cache, and on writes invalidates the key. Only used data gets cached, and a cache outage just makes things slower.

Write-through vs write-behind. Write-through updates cache and database together: fresh reads, slower writes, for balances and inventory. Write-behind acks after the cache and flushes later in batches: fastest, but a cache crash loses data.

Mix patterns per data type. In one store: catalog cache-aside, inventory write-through (or no cache during a sale), prices cache-aside with event-driven invalidation, view counts write-behind. Checkout always reads the database.

TTL is the safety net. Invalidation can fail on a Redis timeout. A TTL sized to the staleness you can accept means a stale value still expires: product details 1 hour, prices 5 minutes, inventory 30 seconds.

• Letting a cache error become a 500 instead of falling back to the database.
• Caching everything: rarely read data evicts the hot set and lowers the hit ratio.
• Collision-prone keys like user; use product:{id}:v1 with a schema version.

Day 2Invalidation Strategies

The cache doesn't know the data changed. A flash sale moved a price from $99 to $79, but the 1-hour TTL kept showing $99 and 147 customers complained. Match each data type's invalidation to its staleness tolerance.

TTL, events, versions. TTL: simple, stale until expiry. Event-driven: publish on change and delete the key in ~100–500 ms, but needs Kafka and a DLQ. Versioned keys: bump product:123:v5 to v6, or a global version to invalidate everything.

Delete, don't update. Write-update races: two writers can leave the cache holding the older value. Deleting is idempotent and the next read loads fresh data; update in place only with one writer or atomic ops like INCR.

Invalidate after commit. Deleting inside the transaction lets a reader cache the old row before commit, leaving cache and database apart indefinitely. Facebook's leases go further: a fill after an invalidation is refused.

Decompose the cache. Cache product:123:details (24 h), price (5 min plus events), inventory (30 s plus events) and reviews (1 h) separately, so an inventory change doesn't evict the description.

• Invalidating only product:123 and forgetting category pages, search results and deals.
• Pattern deletes (KEYS category:*) in the hot path; track keys per category in a set.
• Event-driven invalidation with no TTL: one lost event means stale forever.

Day 3Thundering Herd

One expiring key can take the site down. When a hot key expires, every concurrent request misses and queries the database: 1 miss becomes N queries. 50,000 homepage requests met a database built for 5,000 queries a second.

Coalesce misses into one fetch. Request coalescing (single flight) lets concurrent requests for a missing key join one in-flight fetch and share its result. Instagram caches a promise for the value; locking does the same with more waiting.

Never let hot keys expire. Background refresh recomputes known hot keys (homepage, trending, deals) before their TTL, so users never miss. For a scheduled change, pre-warm the new key and switch atomically.

Serve stale while revalidating. When an entry goes stale, return it immediately and refresh in the background. If the refresh fails, extend the stale window instead of letting it lapse onto a struggling database.

Desynchronize expiry. Jitter TTLs and refresh probabilistically just before expiry, weighted by how long recomputing takes, so keys loaded together don't all expire together.

• One TTL for everything, so the homepage expiring sends 10K queries at once.
• Coalesced waiters with no timeout; fall back to stale data instead.
• Deploying with a cold cache; warm hot keys behind a health-check gate first.

Day 4Feed Caching

Reads vs writes pull apart. Computing a feed on read means querying every followee (500 × 10 ms = 5 s). Precomputing it means every post fans out to every follower: one celebrity post is 50 million cache writes.

Fan-out on write. When a user posts, push the post ID into each follower's feed cache (a Redis sorted set scored by time). Reads become one cache lookup; the cost moves to writes and storage.

Hybrid push and pull. Push posts from regular users (under ~10K followers); store celebrity posts once and pull them at read time, merging with the precomputed feed. Twitter and Instagram run mostly push.

Cache IDs, trim, skip the dormant. Store post IDs, not posts, and trim each feed (~800 entries). Skip long-inactive followers during fan-out and rebuild their feed when they return; fetch pulled posts in parallel.

• Pure push for everyone: one celebrity post backs up the fan-out for an hour.
• Feeds never trimmed, so sorted sets grow forever.
• Fetching each followee's posts sequentially instead of in parallel.

Day 5Multi-Tier Caching

Five layers, five jobs. Browser (static assets, preferences), CDN (public content, media), API gateway (auth validation, rate limits, dedup), Redis (business data, sessions, flags), database buffer pool (automatic).

Invalidate closest to the database first. Delete the Redis key, then the gateway cache, then purge the CDN asynchronously. Purge the CDN first and it may refetch the old value from a still-stale origin cache. Browsers expire only by TTL.

Never let the CDN cache personal data. Mark authenticated responses Cache-Control: private. Vary on headers that change the content (Accept-Language, Accept-Encoding), never on Cookie: that gives every user their own cache entry.

Bounded staleness, not perfect consistency. Design a maximum staleness per tier: CDN 60 s, Redis 5 minutes (usually seconds with events), browser 60 s for API data. Checkout bypasses every cache.

• An authenticated /api/profile cached at the CDN: one user's data served to another.
• Missing Vary: Accept-Language, so Spanish users get the English page.
• Unversioned keys: a deploy reads the old schema; put v3 in the key.

BossDesign a Global E-Commerce Product Catalog System

At 600K req/s, the hit rate is the design. 500M daily views is ~6,000 req/s, and flash sales spike it 100x. Even 1% misses at 600,000 req/s means 6,000 database queries a second: hit rate must exceed 99%.

Freshness per data type. Metadata tolerates an hour (TTL), prices under 5 seconds (events plus a 30 s TTL), inventory under 30 seconds for display, and checkout reads live data. Images can be a week old.

Flash sale protection. Pre-warm the deal before midnight, coalesce concurrent misses into one fetch, refresh hot keys in the background and serve stale while revalidating. Last Black Friday a single expiring key took the site down.

Personalization by activity tier. Precompute recently viewed and recommendations for active users, give recent users shorter TTLs, compute on demand for dormant ones. Caching all 100M users' 3.5 KB would waste most of 350 GB.

Degrade when Redis fails. A circuit breaker detects the failure in under a second; serve stale data from a local in-process cache, hide personalization, pause flash-sale purchases to avoid overselling, then recover half-open after 30 seconds.

• Letting the Deal of the Day key expire at midnight instead of pre-warming it.
• Caching prices at the CDN: checkout then disagrees with the page.
• Precomputing personalization for every registered user, active or not.

Day 1Consistency Models

Who sees a write, and when? A write lands on Node 1 and a read hits Node 2 before replication arrives. Consistency models are precise answers to what that read may return: new value, old value, or an error.

The spectrum, strongest first. Linearizable (behaves like one copy), sequential (one agreed order), causal (cause before effect), read-your-writes, monotonic reads (never go backwards), eventual (converges, no promise when).

Session guarantees users notice. Read-your-writes: route a user's reads to the primary after their write, or track the write version. Monotonic reads: remember the last version read and only accept replicas at or past it.

Choose per operation. Browse products eventually consistent, reserve inventory and check out strongly consistent, order history read-your-writes. DynamoDB and Cassandra let you choose per read: eventual, QUORUM, ALL.

Spanner buys strong consistency with time. TrueTime (GPS and atomic clocks, uncertainty typically under 7 ms) plus Paxos and commit-wait give global linearizability, paid for in write latency: commit-wait plus a Paxos round trip, and more across regions.

• Reading from a replica right after writing to the primary: the user's change "disappears".
• Random replica per read, so a counter appears to go backwards.
• Strong consistency for everything: latency and availability pay for reads that don't need it.

Day 2Distributed Transactions — The Saga Pattern

Why not two-phase commit. 2PC holds locks from PREPARE until the decision, blocks if the coordinator dies, needs every service up at once, and most NoSQL stores, brokers like Kafka and third-party APIs (Stripe) can't take part.

A saga is local steps plus compensations. Each step is a local transaction in one service; if step N fails, run compensations for steps 1 to N-1. No distributed locks, no coordinator holding them.

Compensation is not rollback. A refund is a new transaction that reverses the charge; both stay in history. Compensations must be idempotent, check state first (no-op if the step never ran) and eventually succeed.

The pivot transaction. The point of no return, like shipping. Before it, recover backward (compensate); after it, recover forward (retry until done). In the Airbnb example the pivot is capturing payment.

Choreography vs orchestration. Choreography: services react to each other's events; fine for 2–4 steps but the flow is hard to see. Orchestration: a coordinator calls each step and persists saga state; better for 5+ steps and branching.

• A refund compensation that can fail with no retry: the saga sticks with the customer charged.
• Payment steps without an idempotency key like saga-{id}-payment: a timeout retry charges twice.
• Compensating everything blindly instead of tracking which compensations finished.

Day 3Saga Orchestration Deep Dive

Durable execution. Temporal records every activity result in an event history. After a crash the workflow replays from the start, completed activities return their stored results instead of running again, and execution continues where it stopped.

Workflows decide, activities act. Workflow code must be deterministic because it replays: no random numbers, clock reads, UUIDs or HTTP calls (use workflow.now(), workflow.random(), activities). Activities do the side effects, with timeouts and retries.

Signals and queries. A signal delivers an external event (manager approved, rider cancelled) to a waiting workflow, which can sleep for days at no cost. A query reads workflow state without blocking it.

Version and continue-as-new. workflow.patched("fraud-check-v2") keeps in-flight workflows on the old path while new ones get the change. Long-lived workflows (subscriptions) use continue-as-new so history stays bounded.

• Calling datetime.now() or an HTTP API inside workflow code: replay diverges.
• A workflow that loops forever without continue-as-new: unbounded history.
• Activities without timeouts, so one hung call stalls the workflow.

Day 4Conflict Resolution

Last-write-wins loses data. The highest timestamp wins, so of two concurrent writes one silently disappears, and clock skew can drop the truly latest one. Fine for a location, a profile picture or "last seen"; wrong for carts and counters.

Vector clocks detect concurrency. Each node keeps a counter per node, increments its own on every write and takes the max on receive. If neither clock is ≤ the other in every entry, the writes were concurrent: a real conflict.

CRDTs merge by construction. Merge is commutative, associative and idempotent, so replicas converge whatever the order. G-Counter sums per-node counts; PN-Counter pairs two; OR-Set tags each add so a concurrent add beats a remove.

Pick by data semantics. Latest is right: LWW. Need to detect and ask: vector clocks with siblings (Riak allow_mult). Must merge offline: CRDTs. Collaborative text: OT (Google Docs) or a sequence CRDT (Yjs, Automerge).

• A counter as an LWW value: concurrent increments vanish. Use a PN-Counter.
• A plain set for add and remove: concurrent operations resurrect or drop items. Use an OR-Set.
• Never garbage-collecting tombstones, so CRDT state grows forever.

Day 5Leader Election and Coordination

Timeouts can't tell slow from dead. A GC pause, slow network or clock skew makes a live leader look dead; a follower takes over and two leaders accept writes: split-brain. You can't guarantee both safety and liveness, so favour safety.

Fencing tokens stop zombie leaders. Every election issues a higher token. The leader sends it with each request and resources reject anything below the highest they've seen, so a leader waking from a pause can't corrupt data.

Raft in one breath. Followers expect heartbeats; on an election timeout one becomes a candidate, bumps the term and asks for votes; a majority makes it leader. At most one leader per term, and terms act as fencing tokens.

Don't build it: ZooKeeper or etcd. ZooKeeper's ephemeral sequential znodes make the lowest sequence number the leader, deleted when its session expires; watches announce changes. etcd gives leases on Raft.

Epochs everywhere. Kafka elects partition leaders from the in-sync replicas and bumps an epoch so an old leader is rejected; Redis Sentinel needs a quorum to declare the master down; Kubernetes controllers hold a Lease object.

• A lock with no fencing token: a paused holder wakes up and writes over the new holder.
• "No heartbeat in 5 s, so I'm leader" without asking a quorum.
• Leases on wall-clock time; use monotonic clocks and renew early.

BossMultiplayer Game Matchmaking System

Size it with Little's law. 2M concurrent players, 1.6M in 5v5 matches (160,000 matches of 20 minutes): 8,000 matches start a minute, ~1,300 queue joins a second. At a 30 s wait, only ~40,000 players sit in queue.

One matchmaker per region, fenced. Two matchmakers reading the same queue would put the same 10 players in two matches. Elect one leader per region in etcd, and make every queue change carry its fencing token, checked atomically in Redis.

Ratings need strong consistency. Write ratings to the primary; a player viewing their own rating gets read-your-writes; the queue snapshots the rating from the primary at join time; leaderboards read cached replicas.

Match creation is a saga. Remove players from the queue, ready check, allocate a server, write the match, notify. If allocation fails, compensate by re-queuing confirmed players with priority; a leader crash leaves pending_match_id records to clean up.

Party state as an OR-Set. Kicks, leaves and invites can happen concurrently on different devices. With an OR-Set the merge is deterministic: concurrent removes of Carol both take effect, Dave's add survives. Parties match on their highest-rated player.

• Two matchmakers without fencing: players double-booked into two games.
• Matching on a stale replica's rating right after a match, creating unfair games.
• Removing players from the queue with no compensation if server allocation fails.

Day 1Designing a Real-Time Notification Platform

Two halves. Store-and-forward through providers (push, SMS, email), where "delivered" mostly means a provider accepted it; and a real-time tier where we hold a connection to online users and deliver ourselves in under a second.

Five classes, different clocks. Critical (OTP, fraud), transactional (payments), social, reminder and marketing each have their own latency, channels and quiet-hours rule, and the lane follows. Marketing must never delay a transaction.

Inbox first, then the hint. In-app delivery writes the item to a durable per-user inbox with a sequence number, then sends the gateway a hint. Lose the hint and the client resumes "after 1183". No ack within seconds: push.

OTPs skip the ladder. A code is useless late, so an OTP goes by SMS at once. Security alerts start with push and escalate to SMS if still unread after 2 minutes. Critical ignores opt-outs and quiet hours.

Sent is not delivered. FCM counts a send once the message is "enqueued for delivery". Keep one deliveries row per channel attempt, separate from the notification and the inbox item, and keep sent and delivered apart.

• One queue for every class: the OTP that leaves at 20:08.
• Treating a provider's 200 as delivery.
• The socket as the record: every crash, dropped Wi-Fi and deploy loses messages.

Day 2The Delivery Pipeline: Guarantees, Priorities and Preferences

Key the business event. The idempotency key names the event (payments:pay_8812:received), not the HTTP request, and commits with the notification and its outbox row. A retry gets the original id; a reused key with different parameters gets 422.

Check late, then again. The router reads cached preferences, but the channel worker re-checks opt-out, quiet hours, expires_at and business conditions just before the provider call, marking failures suppressed or expired.

Lanes all the way down. Critical, transactional and bulk each get their own topic, consumer group, workers, retry topics and share of provider quota. One topic with a priority field can't skip a partition's backlog.

At-least-once, deduplicated. Exactly-once isn't available through a provider you don't control. Claim, send, record per task; dedup at every hop; collapse ids hide the rare duplicate. Bulk treats an unconfirmed claim as sent.

Escalation is a forward-only saga. Fraud alert: push now, SMS if unread at 2 minutes, email 10 minutes later, fraud team at 24 hours. Durable timers, not sleep(120); "undo" cancels the steps not yet taken.

• Promising exactly-once instead of at-least-once with dedup.
• Checking preferences only at routing, so opt-outs lag behind the queue.
• Retrying a 410, or delivering an expired OTP.

Day 3Real-Time Channels: WebSockets and SSE at Scale

Pick the transport per surface. SSE for the one-way web bell: free reconnect and Last-Event-ID resume, and HTTP/2 lifts the 6-connection cap. WebSockets for apps that ack. Long polling as the last resort.

Size by connections. 10M sockets heartbeating every 30 s is ~333,000 beats a second against ~1,700 notifications at peak. Cap nodes at 200,000 for blast radius: 50 nodes, 75 to survive a zone.

Registry with leases and epochs. The registry maps user and device to gateway and epoch. Each gateway holds a 15-second lease; lookups skip dead leases and old epochs. One pub/sub channel per gateway.

The socket is a hint. Redis Pub/Sub is at-most-once, which is fine because the inbox already has the item. Clients resume "after seq N" with a range read; no ack within seconds falls back to push.

Heartbeat under every idle timeout. TCP keepalive waits 2 hours; one carrier NAT cut idle TCP at 255 s; an ALB's default is 60 s. Client heartbeat every 25–30 s with jitter; server closes after ~2.5 misses.

Shape the reconnect storm. Full-jitter backoff with a jittered first attempt, connect admission control answering "retry after", resume instead of reboot, bounded per-socket queues, and slow drains: 1% every 6 seconds.

• Using the socket as the record.
• Unbounded per-connection buffers: a few slow phones run the node out of memory.
• Fixed-delay reconnects that recreate the wave every 2 seconds.

Day 4Presence and Chat Fan-Out at Scale

Three classes of event. Durable messages are stored first, ordered and retried. Presence is soft state in memory with a TTL, latest wins. Typing is ephemeral: throttled, expired by the receiver, dropped first.

Presence expires by itself. Heartbeat every d seconds; the entry expires after d + ε, and a delayed trigger at d + 2ε publishes offline only if no newer beat came. Last seen uses the server's clock.

Tell only who's looking. Clients subscribe to the users on their screen. Eager presence costs ~84,000 pushes a second, twice the message traffic; visible-only with debouncing, ~22,000, about half.

Bodies once, pointers to small groups. The body goes once into the conversation log; small groups get a pointer in each member's inbox; huge rooms get no per-member writes, and full live pushes only reach open sessions.

One owner sequences each conversation. The owner assigns 1, 2, 3…, never the sender's clock. A new owner starts from the durable max, and a unique (conversation_id, seq) key fails a stale owner's write.

Receipts are cursors. "Delivered up to 412" and "read up to 410", applied as max(old, new), so ticks never move backwards. A group's tick is the minimum cursor over its members.

• Writing presence to the message database like messages.
• Broadcasting presence to every contact.
• Ordering messages by the sender's timestamp.

Day 5Operations, Scale and Edge Cases

Measure the user's clock. End-to-end latency from created_at, not dequeue time, and the oldest message's age per lane. Success rates only count attempts, so a four-minute queue looks perfect.

SLOs stop at the provider. We can't see an iPhone show a banner. Critical: 99.9% handed off, or acked in-app, within 10 seconds. Bulk has no availability target: pausing a campaign is the system working.

Lanes reach the provider. FCM's quota is per project and an SMS sender has one queue. Reserve quota that bulk can't touch, use separate SMS senders for OTPs and marketing, and short validity periods.

Shift SMS in steps. GOV.UK Notify's failover moved a spike onto a cold provider, which failed too. Always split about 50/50; a 5xx cuts that provider's share 10 points, at most once a minute.

Same code on every attempt. A timeout may have delivered, so failover can send twice. Create the OTP once and key the ledger (notification_id, channel), not by provider, so a duplicate text is harmless.

• Measuring the worker, not the user.
• Counting bad tokens as provider failures, opening a breaker on a healthy APNs.
• No owner for SMS spend: a retry loop or pumping shows up on next month's invoice.

BossFood Delivery Order System

Size the dinner rush. 5M orders a day is ~58 a second; dinner is 4×, ~230. That's ~7,000 writes and ~17,000 browsing reads a second, and ~400K active orders to track.

Never lose an order. Write the order and an outbox row in one transaction; the publisher marks rows published only after Kafka confirms, so a crash republishes and idempotent consumers absorb the duplicate.

Payments: authorize, capture, refund. Authorize at order time, capture when the driver picks up, refund on cancellation, every Stripe call with an idempotency key, so a retried authorization can't create a second PaymentIntent.

No driver assigned twice. Two orders can both see Driver X as free. Assign under a row lock with a version check, so one succeeds and the other tries its next candidate.

Push live tracking, don't poll. Polling 400K orders every 10 s is ~40,000 mostly empty reads a second. One SSE or WebSocket connection per screen: location coalesced to the latest point, status resumed by sequence number.

Order updates through the lanes. Order updates are transactional: inbox first, a live hint, push only if no device acks. OTPs and driver offers are critical; promotions are bulk. Driver online status is presence, never used for correctness.

• Saving the order, then publishing to Kafka: Kafka down means a held card and no food.
• Retrying payment calls without an idempotency key.
• Reassigning an order because a driver's presence dot went grey.

Day 1Search Fundamentals & Architecture

Why LIKE fails. LIKE '%red%' can't use a B-tree, which is sorted by string start, so 50M rows get a full scan. It also has no relevance ranking, typo tolerance, word boundaries, stemming or synonyms.

The inverted index. Pre-compute word to documents instead of scanning documents for words. A query is one lookup per term plus an intersection, smallest list first: "running shoes" takes under a millisecond.

Text analysis. Character filters (strip HTML), then the tokenizer, then token filters (lowercase, stop words, stemming, synonyms). The same analyzer must run at index time and query time, or "Running" never matches the indexed "run".

The multi-field pattern. One source field, several representations: brand as text for search, brand.keyword for facets and sorting, brand.autocomplete for typeahead. IDs are keyword: they're only exact-matched.

Scatter-gather. A coordinating node sends the query to every shard (primary or replica). Each searches its own inverted index and returns its top N with scores; the coordinator merges, sorts by score and returns the overall top N.

The product search cluster. 20 primary shards plus 20 replicas on 6 data nodes, 3 dedicated master nodes, 2 coordinating nodes. Text goes in must to drive the score; category, price and stock go in filter.

• Different analyzers at index and query time, so tokens never match.
• Oversharding: per-shard memory and file-handle overhead degrades queries.
• Not planning primary shards: the count is fixed at creation, so changing it needs _split/_shrink or a reindex.

Day 2Indexing Pipeline

Never dual-write. Writing PostgreSQL then Elasticsearch breaks four ways: the ES write fails, the process crashes between writes, a partition holds the transaction open, or retries reorder updates so ES ends at $100 while PostgreSQL has $120.

Change data capture. App to PostgreSQL; Debezium reads the WAL through logical replication and publishes change events to Kafka; the indexing pipeline consumes them into Elasticsearch. The app writes to one place, and events replay within Kafka's retention.

Version by WAL position. Each bulk operation carries version: event.lsn with version_type: external. If an older change arrives late after retries or reordered batches, ES rejects it with a version conflict instead of overwriting newer data.

Batch, but by priority. Flush at 500 documents or after 5 seconds, so quiet periods don't strand updates. Stock going to zero is CRITICAL in 10-document batches; description edits are NORMAL at 500; bulk imports LOW at 1,000.

Zero-downtime reindex. Reads stay on the products alias over v1 while updates go to both indexes and old data is copied with op_type: create. Verify counts, switch the alias in one atomic remove+add, then delete v1.

• Running a big import through the live pipeline, starving CDC updates.
• Skipping index aliases, so a reindex can't switch with zero downtime.
• Blaming the pipeline when the ES document is right but a result cache is stale.

Day 3Query Processing & Relevance

BM25 in one line. Score grows with term frequency but saturates (k1, default 1.2), rare terms weigh more (IDF), and long documents are normalized (b, default 0.75). 10 occurrences aren't 10x better than 1.

Rare terms win. In the example corpus, "nike" (2% of documents) has IDF 3.912, "shoes" (10%) 2.3026, and "the" (90%) 0.1054. A match on the rarer term says more about relevance.

Query context vs filter context. What the user typed is scored in query context. What they selected (facets, toggles, ranges) goes in filter: yes or no, unscored, cacheable as a bitset. The lesson puts the split at ~3x faster.

Boost in layers. Start with field boosts (name^3, brand^2, description), add business boosts in should clauses, and use function_score (popularity, recency decay, rating, featured) only when needed. A/B test every ranking change.

Never return nothing. Progressive relaxation: exact AND query, then drop low-priority filters (rating, price), then fuzzy matching, then any-term matching, then no filters at all. Category stays: it's a strong intent signal.

Pick the multi_match type. best_fields with tie_breaker 0.3 is the product default. cross_fields lets "John" match first_name and "Smith" last_name. most_fields sums scores across analyses. phrase_prefix and bool_prefix serve autocomplete.

• Putting category, brand and price in must: they sway scores without adding relevance and aren't cached.
• Filtering out-of-stock items away when they should be demoted with a boosting query.
• Over-boosting, which makes relevance unpredictable.

Day 4Typeahead at Scale: Autocomplete, Spelling and Facets

Typeahead multiplies traffic. One request per keystroke: a search is ~12 requests. The store's ~2,800 searches a second at peak become ~33,000 typeahead requests, so typeahead is its own cheap system, not prefix queries on the search cluster.

Store the answer, not the means. A plain trie ranks whole subtrees, largest for short prefixes. Precompute each prefix's top-k at build time: serving is one walk or map lookup, O(prefix length). An FST is the compressed, immutable version.

Suggestions come from query logs. Normalize, count searches and distinct users, decay with a half-life (14 days, illustrative), filter, then ship a versioned snapshot nightly. A stream job adds a small trending overlay, capped in slots and filtered more strictly.

Suggestions are the system speaking. A distinct-user minimum keeps one person's phone number out. PII patterns, blocklists and policy lists drop the rest, including unconfirmed rumours. A serve-time blocklist plus CDN purge removes misses in seconds.

Global at the edge, history private. The global list is the same for everyone per prefix and locale, so the CDN caches it on that key. History comes privately, merged on the client. Never cache a blended list.

After Enter. Spelling: pick the c maximizing P(c) x P(w | c), with your query logs as the language model. Facets: selections in post_filter, each facet filtered by every selection except its own, so Adidas stays countable.

• A prefix query on the search cluster per keystroke: a sale night slows search for everyone.
• Raw all-time counts: old models win forever and one user's query leaks.
• Sharding by first letter: skew, and launches make one shard hot.

Day 5Location Search and Search at Scale

Why a B-tree can't do "near". An index on (lat, lng) uses only the latitude range, then tests longitude on every row: a strip across the country. Instead, cover the query with cells, then refine by exact distance.

Geohash and its edge. Each base-32 character adds 5 bits and divides the cell by 32. A shared prefix means close, not the reverse: points 43 m apart can share only tdr. Search the cell plus 8 neighbours, then refine.

S2 and H3. S2 numbers a quadtree on a cube in Hilbert order: a cell's descendants are one ID range, ideal for coverings and shard keys. H3 hexagons have six equidistant neighbours, so rings approximate circles; children don't nest exactly.

Shapes in a database. PostGIS puts an R-tree on GiST that stores bounding boxes; ST_DWithin uses it, ST_Distance in WHERE doesn't. Delivery zones are polygons: ask which zones contain the customer's point.

Moving objects are soft state. Overwrite each partner's position in memory on every ping; touch the cell index only on a cell change. A TTL drops dead phones. The trail goes to Kafka, never into the search index.

Stop kNN with a bound. Grow H3 rings outward, but stop only when you have k results and the k-th is no farther than a lower bound on anything unscanned. Re-rank the top few by road ETA.

• Updating a search document on every ping: reindexing and segment churn slow product search.
• Trusting the prefix: missing a partner 43 m away across a cell edge.
• One shard per city: one stadium slows the whole city.

BossDesigning a Legal Document Search System

Size the corpus. 100M documents (50M public, 50M private) at ~65 KB is 6.5 TB of text. 2x index overhead and 2x for replicas make 26 TB in Elasticsearch, plus 50 TB of original PDFs in S3: ~76 TB.

Size the traffic. 50,000 concurrent users at 20 searches an hour is 1M an hour, ~280 a second; autocomplete at 5 keystrokes per search is ~1,400 a second. Today's 10,000 a minute (~170/s) leaves ~1.7x headroom.

A real query parser. A lexer and parser turn (breach /5 contract) AND damages NOT punitive into must, should and must_not clauses. Proximity becomes span_near with in_order: false; /s means ~15 words, /p ~50.

Isolation in layers. Per-tenant indices, a service-injected tenant_id filter, document-level security, a tenant taken from the validated JWT (never the request body), and audit logs. Zero tolerance means no single control is trusted alone.

Citations as edges. A separate index of {citing_id, cited_id, treatment} makes "who cites X" a term query on cited_id, with aggregations for treatment breakdowns, "good law" checks and 2-hop chains.

Triage every upload. Store the original in S3 first. A PDF with under 100 characters of text in its first pages is scanned: render each page at 300 DPI and OCR it. Word goes through Tika. Then publish to Kafka.

• Trusting one isolation control, or a tenant_id from the request body.
• Using a phrase query for proximity: "liability for negligence" should match negligence /10 liability.
• Fuzzy matching citation variants instead of a canonical case ID plus name synonyms.

Day 1Event Ingestion at Scale

Events are immutable facts. Each event has a unique event_id, an event time, a type and context. Never update one: record the change as a new event (event.processed) pointing to the original, or replay and the audit trail break.

Evolve schemas safely. A registry enforcing BACKWARD compatibility rejects a new required field like session_id, since old events lack it. Add fields as optional with a default. Name events domain.entity.action, in the past tense.

Collect, validate, then Kafka. Lightweight collectors validate size, required fields, schema, timestamps, business rules and security, then write to Kafka asynchronously. In a batch, valid events go to Kafka and invalid ones to the dead letter topic.

Timestamps and enrichment. An event time over 5 minutes in the future is an error; over 30 days old is only a warning, since offline phones send those. Enrichment (geo, user, product, session) is best-effort: a failed lookup never blocks.

Key by user, write idempotently. Kafka orders only within a partition, so user events are keyed by user_id. acks=all makes a write durable across in-sync replicas; enable_idempotence stops a retried send from writing a duplicate.

Shed telemetry, never revenue. Above 90% queue usage the collector sheds page.viewed-type telemetry but never orders, payments or sign-ups. Per-client limits (1,000 a second) and an adaptive limiter protect downstream systems.

• Events without an event_id: no deduplication, no tracing through the pipeline.
• Writing to the database before replying: a slow database makes a slow API.
• No dead letter queue, so a failed event is lost forever.

Day 2Streaming vs Batch Processing

Two ways to process. Batch processes large chunks: hours of latency, exact, cheaper, simpler. Streaming processes each event as it arrives: seconds of latency, approximate because late events may be missed, always on and more complex.

Lambda. Every event feeds a Spark batch layer (accurate, overwrites its views) and a Flink speed layer that fills the gap since the last batch; the serving layer adds them. Its big problem: two codebases that drift apart.

Kappa. One streaming codebase over a long-retention log. To reprocess, start a new version from the earliest offset, let it catch up, switch and delete the old output. 1B events at 100K/s replay in about 3 hours.

Watermarks. Watermark = max event time seen minus max out-of-orderness. With a 5-minute bound and latest event at 10:02 it's 9:57, so [9:00-10:00) fires only after an event at 10:05 or later. Wall-clock time plays no part.

Late data and state. An event after its window fired can be dropped, side-output, refire the window, or fit within allowed lateness. Keep large state in RocksDB with a TTL, and checkpoint so a crash doesn't start over.

Write once, run both. Apache Beam, Flink (batch is just bounded streaming) and Spark Structured Streaming run the same aggregation as a stream or a batch, so real-time and batch results can't drift apart.

• Writing batch output with mode("append"), so a retried job duplicates rows; overwrite the date partition.
• Keeping state forever with no TTL, until it runs out of memory.
• Grouping by a skewed key without salting, so one task runs for hours.

Day 3Data Modeling for Analytics

OLTP vs OLAP. OLTP runs the business: many small transactions, normalized (3NF), row storage. OLAP understands it: few large queries over many rows and few columns, denormalized, column storage. "Revenue by country" becomes one scan and a GROUP BY.

Star schema. A fact table of measures (revenue, quantity) at a stated grain, here one row per line item, with foreign keys to denormalized dimensions: product, customer, date, store. order_id is a degenerate dimension kept in the fact.

SCD Type 2. On a change, close the old row (valid_to, is_current = FALSE) and insert another. Facts store the surrogate key valid at sale time, so a plain join is point-in-time. Only tracked columns create versions.

Load dimensions, then facts. Dimensions first, so today's new versions have surrogate keys. Lookups use the version valid on the batch date, not is_current, so backfills stay right. A missing product maps to unknown key -1, keeping totals whole.

Columnar storage. SUM(revenue) over a 4-column table reads 1 column: 75% less I/O, better compression, vectorized. Parquet row groups carry statistics for predicate pushdown; Delta Lake and Iceberg add ACID transactions and time travel.

Partition and pre-aggregate. Partition by filter columns like date, Z-order by a second filter such as product_id, never partition by a high-cardinality key. A daily rollup by category scans 365 × ~100 = 36,500 rows instead of 1B.

• An order-level grain: you can never drill down to products.
• Business keys in the fact table instead of surrogate keys, so dimension history is lost.
• Partitioning by customer_id: 10 million partitions full of small files.

Day 4Late-Arriving Data and Correctness

Event time, not processing time. A purchase made at 14:30 on the subway reaches the server at 15:45, 75 minutes late. Windowed by processing time it lands in 15:00-16:00; by event time it lands in 14:00-15:00, where it happened.

Two separate knobs. Watermark = max event time minus the out-of-orderness bound: with a 5-minute bound and max 10:03 it's 9:58, and an older event can't lower it. Allowed lateness is separate: how long a fired window still accepts updates.

Per-key watermarks idle out. With per-key watermarks the global one is the minimum, so a phone offline for two days pins it in the past and windows stop closing. Keys not seen within the idle timeout are left out.

Tune lateness from data. Measure the lateness distribution (example: p50 2 seconds, p99 5 minutes, max 72 hours) and set it per source: minutes for web, hours for mobile. Unlimited lateness grows state until it runs out of memory.

Patch or reprocess. Patch additive metrics (SUM, COUNT) when late data is small. Reprocess from raw events, overwriting the range, for non-additive ones (AVG, MEDIAN, DISTINCT), big late volumes or changed logic.

Version, label, finalize. Store every computation as a version with a reason and delta, so latest, final and as-of queries work. Stream results are provisional; a daily batch job rereads raw events for final numbers. Label dashboards Live or Final.

• Windowing by processing time, so offline events land in the wrong hour.
• Overwriting results in place, leaving no answer to "why did yesterday change?"
• Alerting on live numbers; gate the normal threshold on is_final, a stricter one for live.

Day 5Query Layer and Optimization

Match the engine to the access pattern. Cloud warehouses (Snowflake, BigQuery) answer in seconds, managed, for ad-hoc SQL. Real-time OLAP (ClickHouse, Druid) gives milliseconds and high concurrency for user-facing dashboards. Trino queries the lake in place; DuckDB runs in-process.

Why the dashboard is slow. "Revenue by category, region and day for 90 days" scans ~2.7B of ~11B rows (30M a day), shuffles 50GB and takes 47 seconds: no partitioning, no pre-aggregation, no caching, joins that shuffle.

Partition, then cluster. Partitioning decides which files to read, and only helps when the query filters on the partition column. Clustering (sorting, Z-ordering) decides how rows sit inside files, so category = 'A' can skip files without it.

Pre-aggregate the predictable. Pre-aggregate patterns that are predictable and frequent (over 10 a day): the executive dashboard drops from 45 seconds to 50ms. Route to the coarsest rollup that keeps the needed granularity; leave rare ad-hoc slices on demand.

Cache with the right TTL. Layers: CDN, Redis query results (60-80% hits for popular queries), database cache, block cache. TTL follows freshness: about a minute for today, hours for history. Include the tenant in the key: same SQL, different rows.

Isolate tenants' data and compute. Row-level security is cheap, but one missing filter leaks data. Per-tenant quotas (10 concurrent queries, 100GB scanned a day, 300-second timeout) stop one noisy tenant from slowing everyone else.

• Caching everything for 24 hours, so live panels show stale numbers.
• One agg_everything rollup grouped by every dimension: nearly as big as the fact table.
• No query timeouts or byte limits, so runaway queries block others and costs explode.

Day 6Counting at Scale: Bloom Filters, HyperLogLog, Count-Min Sketch and Top-K

Counting means remembering. Exact distinct counts cost memory per distinct key, per slice, per day: 10M visitors in 3 of 10,000 cells is ~1.2GB a day. Sketches cost a fixed amount with a known, bounded error.

Bloom filter: "no" is certain. Set k bits per key; any 0 bit means never inserted. False positives only: ~9.6 bits per key at 1%. Over-fill silently ruins it: 3x the planned keys pushes 1% to ~44%.

HyperLogLog counts distinct. Registers keep the longest run of leading zeros in hashed ids. Error is 1.04/√m: 0.81% in 12KB at p = 14. Re-adding an id changes nothing, so replays don't inflate it.

Merge sketches, never sum uniques. HLLs merge by register-wise max into the sketch of the union. Store one per day and slice; any range is a merge. Summing daily uniques counts visitor-days, not visitors.

Count-Min for heavy hitters only. d rows of w counters; read the minimum. Never under-counts, over by at most εN of the whole stream. At ε = 0.001 over 50M views, a 2,000 estimate means anything from 0 to 2,000.

Top-k with Space-Saving. Keep m counters; a new key evicts the minimum and inherits it as its error, so counts over-estimate by at most N/m. Key by item, one summary per minute pane, merge the last 60.

• Dropping events a Bloom filter "has seen": 1% of new events silently lost. Verify "maybe" exactly.
• Adding daily or regional unique counts instead of merging sketches.
• Merging per-partition top-10 lists: an item 11th everywhere vanishes.

BossReal-Time Analytics for Video Streaming Platform

Size the firehose. 10M peak viewers sending one event every 10 seconds is 1M events a second; a day adds up to ~37B, planned as 50B. Heartbeats are 70% (35B). At ~300 bytes that's 15TB a day raw, 3TB compressed.

Two serving stores. Regional collectors write to Kafka (events.playback keyed by user_id, 7-day retention). Flink feeds ClickHouse for trending and live counts; Spark feeds BigQuery for history and A/B tests; raw events land in S3.

Acknowledge after Kafka. Collectors validate and batch, but answer the client only after Kafka acks: acking earlier would lose buffered events if a collector crashed. 20 collectors per region handle ~10K events a second each.

Trending from unique viewers. Raw counts favour long content and bots, so score unique viewers (HyperLogLog) in a 1-hour window sliding every minute, normalized by a category baseline and boosted by velocity. Scores refresh every 30-60 seconds.

Three freshness tiers. Events under 5 minutes late update real-time numbers; under 24 hours they queue for the T+1 preliminary job at 6 AM; later ones wait for the T+7 final job. Phones persist events in SQLite.

Creator analytics. A star schema (fact_daily_views with five dimensions) partitioned by viewing date and clustered by content. 90% of queries hit the agg_daily_content rollup; HLL sketches merge uniques across days instead of summing them.

• Acknowledging clients before Kafka acks, so a collector crash loses events.
• Counting raw events for trending, which favours long titles and bots.
• Summing daily unique viewers over 30 days, so a 5-day viewer counts 5 times.

Day 1Tenant Isolation Strategies

The isolation spectrum. Shared tables with a tenant_id column, then a schema per tenant, a database per tenant, dedicated infrastructure and separate cloud accounts. Each step adds isolation and cost; no tenant identifier at all is never acceptable.

Tenant from the token. The verified JWT claim is authoritative; a subdomain or X-Tenant-ID header is only a hint and a mismatch is rejected. The tenant lives in a ContextVar, so concurrent requests can't see each other's tenant.

Fail loudly without a tenant. get_current_tenant() raises TenantContextError when no context is set. Any fallback, a default tenant or the previous request's, is exactly how a query ends up without its tenant filter.

RLS as defense in depth. Application repositories add tenant_id to every query; PostgreSQL RLS filters again in the database. FORCE makes policies apply to the table owner too, and app.current_tenant is cleared before a pooled connection is released.

Context must travel. Events carry tenant_id (also the partition key) and jobs take it as an explicit argument, because a ContextVar doesn't reach another process. Cache keys, file paths, search indexes, logs and metrics all include the tenant.

• A query without WHERE tenant_id = ?, returning every tenant's rows.
• Trusting a client-supplied X-Tenant-ID, so an attacker picks the victim's tenant.
• Cache keys like user:{user_id} without a tenant prefix: IDs repeat across tenants.

Day 2Noisy Neighbor Prevention

One tenant breaks everyone. One "export all data" report scanned 2.3 billion rows for 847 seconds: API latency went from 200ms to 15,000ms and 500 other tenants couldn't use the product. Legitimate use, no limits.

Rate limits aren't enough. Rate limiting stops request volume, not expensive operations: five requests a minute can still scan billions of rows. Add concurrency, compute and volume quotas, and enforce at the gateway, application, database and infrastructure.

Limits shared across servers. A token bucket refills up to burst_size: short bursts pass, the long-run rate holds. Per-server limiters give a tenant N times its limit, so one atomic Lua script updates a Redis sliding-window log.

Guard the database. Concurrency slots expire after their TTL, so a crashed worker can't leak one forever. EXPLAIN-based analysis rejects queries over 10 million estimated rows unless the plan is Enterprise; per-tenant pools set statement timeouts.

Schedule fairly. Round-robin across per-tenant queues: a tenant with 100 queued jobs can't make another wait for all of them, and each queue is capped. Weighted fair queuing (free 1, pro 5, enterprise 20) gives paying tiers a larger share.

• One global limiter: a tenant at 900 req/sec leaves 100 for everyone else.
• Throttling silently by sleeping, which hides the limit and holds connections; return 429 with Retry-After.
• One limit for every operation: 1,000 exports melt the database.

Day 3Data Residency and GDPR

Four words to keep apart. Residency is where data is stored; sovereignty is which country's laws apply; localization is a legal requirement to keep data in-country; a transfer is moving data across borders, which needs a legal basis.

What counts as personal data. Names, emails, IP addresses, cookie and device IDs, location and behaviour. Pseudonymized data is still personal data, and backups and caches follow the same rules as the primary data.

Regional data, global control plane. The common SaaS pattern: each tenant is assigned a region at onboarding, and its database, storage and search live there. The global control plane holds only metadata: tenant registry, configuration, routing rules.

Consent done right. Freely given, specific per purpose, informed, unambiguous, and as easy to withdraw as to give. Keep an append-only log (exact text, version, IP, time) plus a current-status table; never asked counts as not granted.

Transfers need a basis. Adequacy decision first, then Standard Contractual Clauses with supplementary measures, or BCRs for corporate groups. Post-Schrems II, document a Transfer Impact Assessment and review it, because adequacy can be revoked.

• Keeping EU data in Frankfurt but sending backups to a US bucket.
• Logging emails or user IDs to a global logging service.
• Using US vendors without a DPA and a transfer mechanism, or assuming adequacy is permanent.

Day 4Right to Deletion

What Article 17 asks. Respond within one month (two more for complex cases), delete every copy including backups eventually, and have processors delete too. You may refuse for legal obligations, public interest or legal claims, and must explain why.

Deleting is not marking. Soft delete leaves the data in storage and backups, so it's only a step before hard delete. Pseudonymized data is still personal; truly anonymous aggregates ("users made 10,000 orders") may be kept.

Inventory, then orchestrate. Map every place personal data lives, then run one executor per system in dependency order: search, files, analytics, third parties, the primary database (it holds the user_id mapping and foreign keys), and caches last.

Stop, retry, verify. Targets at the same priority run in parallel; after a failed group the run stops and a retry resumes at that point. The request completes only after every system is re-checked and the data is confirmed gone.

Keep what the law requires. Orders are financial records, so they're anonymized (user_id NULL, addresses 'DELETED'), not deleted. Shared content shows "Deleted User". The deletion audit trail is kept, with the user ID redacted, as proof.

Backups expire, restores re-delete. You can't remove one user from a backup, so backups expire under a retention policy that's tracked. If a backup is restored first, the pending deletions are re-applied before the data goes live.

• Deleting the users row and forgetting search, analytics, caches and third parties.
• Reporting "deleted" without a verification pass over every system.
• Deleting audit logs with the user, leaving no proof of what happened.

Day 5Security Architecture

Defense in depth. Five layers, each assuming the others might fail: perimeter (WAF, DDoS protection, rate limits), network (VPC, private subnets), application (authn, authz, validation), data (encryption, keys), monitoring (audit logs, anomaly detection).

Zero trust. Never trust, always verify: network location grants nothing, every connection is authenticated and every request authorized. Least privilege, assume breach, micro-segmentation. Google built BeyondCorp after Operation Aurora.

Secrets never in code. Not in code, config files, container images or environment variables. Fetch from Vault or Secrets Manager at connection time so rotation needs no restart. A leaked secret is already copied everywhere: rotate it.

Encrypt at every layer. TLS 1.2+ externally and mTLS between services; TDE and SSE-KMS at rest; field-level encryption for SSNs and cards, because TDE decrypts for any query. A key hierarchy (HSM master, tenant keys, data keys) limits blast radius.

Revocable tokens. 15-minute access JWTs and 7-day refresh tokens, plus a server-side session checked on every request, so deleting the session revokes access at once. Five failed logins mean a 15-minute lockout; errors stay generic.

• Trusting frontend validation: amount=-500 sent straight to the API adds money.
• Fetching documents by ID alone, without AND tenant_id = ? from the authenticated user.
• allow_origins=["*"] with credentials, or logging passwords, card numbers and auth headers.

BossDesign a Multi-Tenant Enterprise SaaS Platform

Size it first. ~100M documents at 500KB is 50TB; with metadata 60TB, 180TB with 3x replication. Extracted text of 10KB each is a 1TB index, ~3TB in Elasticsearch. 1,000 requests a second at peak, 60% of them searches.

Regional data planes, global control plane. US, EU and APAC each run a full stack: API, PostgreSQL, Elasticsearch, S3, Redis, Kafka. The global control plane holds only the tenant registry, flags, quotas and routing. No tenant data crosses regions.

Five isolation layers. Tenant from the verified JWT (the subdomain must match); repositories that add tenant_id; RLS in PostgreSQL; a search index per enterprise tenant or a filtered shared index; S3 paths and IAM by tenant, with per-tenant keys.

Quotas and fair queues. Per-plan quotas on API calls, searches, uploads and query timeouts. Each tenant gets its own capped job queue; dividing the plan weight by (1 + depth * 0.1) shrinks a deep queue's share.

Uploads go straight to S3. The upload service issues a presigned URL for the tenant's regional bucket, so bytes skip the API servers; text extraction and thumbnails run asynchronously from a queue. The server picks the bucket, never the client.

• Reading tenant_id from the request body so admins can "switch tenants".
• Putting user emails in the global control plane for faster login routing.
• Deleting the primary database first, leaving search, file and analytics copies you can no longer find.

Day 1SLIs, SLOs and SLAs

Measure, target, promise. An SLI is the measurement (share of requests under 200ms), an SLO the internal target (99.9% over 30 days), an SLA the promise with consequences (99.5%, or a 10% credit). SLO stricter than SLA.

Specify the SLI precisely. Good events over valid events, measured at the load balancer for the client's view. Exclude health checks and planned maintenance; count 4xx as good, because the service handled them correctly.

Appropriate reliability. 100% is impossible, infinitely expensive and leaves no room to ship. Going from 99.9% to 99.99% may cost ~10x infrastructure and slow features, yet most users can't tell. Don't set the SLO at current performance: leave headroom.

Error budgets drive decisions. 99.9% over 30 days allows 43.2 minutes of failure. Above 50% left, ship freely; 20-50%, no risky deploys; under 20%, pause features; exhausted, freeze. The policy must be written down and enforced.

Alert on burn rate. Burn rate is the actual error rate over the allowed one; 1x empties the budget exactly at the window's end. Page only when a short and a long window both exceed it, so blips don't page.

• SLOs on server metrics like CPU < 70%: requests can fail at 50% CPU.
• Setting the SLO at current performance, so normal variation keeps breaching it.
• Fifty SLOs instead of 3-5 user-facing ones, or SLOs with no error budget policy.

Day 2Observability

Monitoring vs observability. Monitoring watches for known failure modes with pre-defined dashboards and alerts: it tells you when. Observability lets you ask questions you didn't anticipate, like slicing slow requests by app version: it tells you why. You need both.

Three pillars, one trace ID. Metrics say when and how bad, logs what happened, traces where in the request path. A trace ID propagated through every service and written in every log, plus exemplars on metrics, turns hours into minutes.

Pick the right metric type. Counters only go up (use rate), gauges go up and down, histograms count observations in buckets. Histograms add up across pods for a fleet-wide p99; summaries pre-compute per-pod quantiles that can't be aggregated.

Keep labels bounded. Total series is the product of label cardinalities: method × endpoint × status is 10,000 series, but adding user_id for 1M users makes 10 billion. Per-user detail belongs in logs and traces.

RED for services, USE for resources. Rate, errors and duration for APIs; utilization, saturation and errors for databases, queues and disks. Dashboards show the four golden signals first and compare with yesterday and last week.

Propagate the context. Service B continues A's trace only if A injects the context (traceparent or B3 headers) and B extracts it. Otherwise each service starts its own trace and the journey is lost.

• Alerting on causes like CPU > 80% instead of symptoms users feel, such as p99 latency.
• Logs without context ("Payment failed") or with secrets: no passwords, full cards or tokens.
• Alerts without runbooks, or error_count > 0, which is pure noise.

Day 3Deployment Strategies

Small and frequent. The safest deployment is a small one; the safest rollback is a fast one. Deploy daily in small increments, early in the week, watch dashboards during the deploy, and declare success only after the bake time.

Rolling and blue-green. Rolling replaces a few instances at a time: no downtime, but slow rollback, and both versions share the database and APIs. Blue-green switches all traffic at once; rollback is instant because the old side still runs.

Canary with automated analysis. Shift 1%, 10%, 50%, then 100%, watching each stage for 15 minutes. Roll back if errors rise over 1% absolute or p99 over 10% versus stable; under 100 requests, fail closed, never promote.

Decouple deploy from release. Ship code with the flag off, then enable it gradually. Hashing flag_key:user_id into a bucket 0-99 keeps each user on one path. Remove release flags after rollout; keep kill switches.

Expand, migrate, contract. To rename a column: add the new one and write both; backfill, then switch reads; drop the old column only when no running code uses it. Old and new code run together during every deploy.

• Renaming a column in place while old and new code both run.
• Schema and code changes in one deploy, so neither can roll back alone.
• No rollback plan, or a deploy on Friday followed by walking away.

Day 4Capacity Planning

The capacity cycle. Measure what you can handle now, forecast demand (growth, seasonality, events), find what breaks first, plan the scaling, validate with load tests, then measure again. Scale before you need to: projects take 4-6 weeks.

Headroom. Headroom = (max capacity − current) / max capacity. A database good for 3,000 QPS at 2,000 has 33%. Keep at least 30% on critical resources: traffic is bursty, retries storm, GC pauses happen, and 100% means already failing.

Capacity is the sustainable max. Stress-test the latency curve: in the example, latency stays within the SLO up to 1,000 RPS and errors start at 1,800. Capacity is 1,000, not the breaking point; run at most ~700.

Little's Law. L = λ × W. 500 RPS at 200ms needs 100 concurrent workers: 25 pods at 4 workers each, ~36 with 30% headroom. Past capacity, requests queue, W grows, L grows, and latency spirals.

Stay below 70-80%. In a single-server queue, wait ≈ service time × ρ / (1 − ρ): about 1x at 50% utilization, 4x at 80%, 9x at 90%, infinite at 100%. Never run critical resources above 70-80%.

Pre-scale known events. Auto-scaling takes 3-5 minutes, so a sudden spike lands first. Pre-scale for launches and campaigns, scale up fast and down slowly (+3 pods vs −1 with a 15-minute cooldown), and plan for peaks, not averages.

• Waiting for problems: by the time you see them, users are affected.
• Trusting averages: 30% average CPU can hide an evening peak near 95%.
• Load testing only in staging, or ignoring the database and third-party limits behind you.

Day 5Incident Management

Severity sets the response. SEV1, complete outage over $10,000 an hour: all hands, execs, status page. SEV2, a major degradation: on-call plus backup. SEV3 waits for business hours; SEV4 goes to the ticket queue. Not every alert is an incident.

Mitigate first. Detect, triage, respond, mitigate, resolve, learn. Stop user impact before chasing the root cause: recent deploy, roll back; dependency down, fail over or degrade; overloaded, scale up or rate limit.

Clear roles. The Incident Commander coordinates and decides but doesn't debug; the Technical Lead diagnoses and fixes; the Communications Lead runs updates and the timeline. One change at a time, announced first.

Communicate on a clock. First status page update within 10 minutes; SEV1 updates at least every 15 minutes, even "still investigating". Plain, impact-focused language; never blame a provider you chose.

Blameless postmortems. Within 48 hours. Ask why the system allowed it, never stop at human error, and use the five whys until you reach something fixable. Action items get owners and dates and are tracked like any work.

Sustainable on-call. At least 8 people in rotation, fewer than 5 pages per week-long shift, handoff meetings between shifts, and only actionable alerts with runbooks. 50 noisy pages a week train people to ignore the real one.

• Too many cooks: several people changing things at once.
• Going silent while heads down, so customers learn from Twitter.
• No postmortem, or one that stops at "the engineer forgot to test".

BossThe Grand Finale

Size the bidding platform. 10 billion requests a day is ~116,000 a second, ~500,000 at peak. At ~8 KB of audit data each: 80 TB a day, 29 PB a year, ~200 PB over 7 years, so tiered storage.

A 100ms latency budget. Receive 2ms, parse 5ms, DSP fan-out 75ms, auction 10ms, respond 3ms: 95ms, with 5ms to spare. The 70ms DSP timeout is a hard deadline inside the fan-out; a late answer loses the impression.

Budget leases, never overspend. A global coordinator splits each budget across regions, keeping a reserve; regions lease small amounts to pods with an atomic Redis check-and-decrement. Pods spend locally, so no network call per bid; no lease means no bid.

Fail toward under-spending. If a region fails, its leases expire after 30 seconds and only provably unspent budget is released; the rest stays locked until reconciliation. In a partition a region bids on existing leases, then stops.

Fan out with breakers. Query 5-10 DSPs in parallel under one global deadline and auction whatever arrived. A per-DSP breaker opens after 5 consecutive failures and retries after 30 seconds. Under overload, query the top DSPs and shed low-value requests.

Audit through an outbox. Insert the bid event into an outbox before replying (one local write); Debezium ships it to Kafka keyed by advertiser_id for ordered billing. Hot 7 days, warm Parquet to 90 days, Glacier to 7 years.

• Each server checking "is there budget?" alone: $10 left becomes $19,990 overspent.
• Publishing audit events straight to Kafka beside the budget write: a dual write can lose them.
• Waiting on slow DSPs instead of one hard deadline and circuit breakers.

Day 1LLM Serving: Tokens, Latency and Cost

Size it in tokens, not requests. A model call's work, price and rate limits scale with tokens in and out. 900 turns a minute sounds small; at ~3,835 input tokens a turn it's 3.45M input tokens a minute.

Prefill, then decode. Prefill reads every input token in one parallel, compute-bound pass and sets time to first token. Decode writes one token per step, bound by memory bandwidth, and sets time per output token.

Output length is the latency lever. Latency = TTFT + TPOT × output tokens. A 250-token reply at 20 ms a token is ~5 s of decode. Halving output may cut ~50% of latency; halving the prompt only 1-5%.

Tokens resident set concurrency. The KV cache holds every token of every running sequence in GPU memory. A 20,000-token prompt holds the memory of five 4,000-token ones, so long prompts cut concurrency and raise TTFT.

Errors arrive after the 200. Stream end to end: no buffering proxies, SSE comment heartbeats under idle timeouts, cancel on disconnect. An overloaded_error can arrive mid-stream: tell the client in-band, store the reply as interrupted.

Admit by tokens in a gateway. One gateway counts tokens, keeps shared buckets for input and output, reserves a share for interactive chat, queues with deadlines, and retries 429 and 529 only before the first byte, with retry-after and jitter.

• Sizing by requests: 900 a minute is 3.45M input tokens a minute.
• Treating 200 as success, or saving a partial reply as the answer.
• Retrying 429s at once, after the first byte, or on a spend-cap 429.

Day 2Embeddings and Vector Search

Keywords miss meaning, vectors miss tokens. "paisa wapas" needs an embedding to find the refunds article; "RTN-4471" needs BM25, because one return code embeds much like another. Use the same embedding model and version for queries and documents.

Exact search is a bandwidth problem. 3M vectors × 4 KB is 12 GB per query, ~0.6 s at 20 GB/s. Keep exact search for ground truth and selective filters: one customer's 2,000 tickets scan faster than any index.

HNSW, a graph you descend. Sparse top layers make long greedy jumps; layer 0 holds every vector and keeps a candidate list of size ef. Raising ef_search buys recall with latency. Deleted rows linger until a vacuum or rebuild.

IVF and product quantisation. IVF probes nprobe of nlist k-means clusters; cheap appends, but centroids drift. PQ stores a 1,024-dim vector as 64 one-byte codes, 64× smaller; rescore the top 100-200 with full vectors.

Filter inside the search. pgvector post-filters approximate scans: ef_search 40 at 10% selectivity returns about 4 rows, silently. Use iterative scans, partial indexes for a few known values, or an exact scan for tiny sets.

Fuse ranks, not scores. BM25 is unbounded and query-dependent; cosine sits in a narrow band. Reciprocal rank fusion sums 1 ÷ (60 + rank) per retriever, so agreement wins. Pass the reranker a generous candidate list.

• Vector search alone, so order codes and SKUs go missing.
• Filtering after the index and getting 3 results instead of 10, with no error.
• Querying with a new embedding model against vectors from the old one.

Day 3RAG Systems

The answer is only as fresh as the index. RAG is a search pipeline with a model at the end. Lewis et al. swapped 2016 and 2018 indexes and accuracy collapsed; MegaMart's stale policy was a hot-swap that never happened.

Chunk along structure. Parse tables with their headings, chunk one section at a time with document and section titles inside, and dedup by content hash. Contextual retrieval prepends 50-100 tokens of document context to each chunk.

Freshness is an SLO. CDC keyed by doc_id, idempotent upserts on doc_id:version:chunk_no, deletes as events, a dead-letter queue for poison documents, a reconciliation sweep, and a canary measuring publish to searchable.

Retrieve wide, rerank narrow. Hybrid search returns ~150 candidates; a cross-encoder reads query and chunk together and keeps ~8. Below a score threshold, abstain and offer a human rather than answer from weak context.

Retrieval runs as the user. Filter by tenant and ACL inside retrieval, so unauthorised chunks never reach the reranker, the model or the logs. A prompt saying "don't reveal" isn't access control. Entitlements down: public content only.

• Tuning the prompt before reading the retrieved chunks.
• Fixed-size chunks that cut tables from their headings.
• Permissions in the prompt, or a filter applied after the top-k.

Day 4Evaluation and Guardrails

Quality is measured, not unit-tested. Outputs vary between runs, every prompt change is global, and wrong answers look like successes. Gate changes on a versioned golden set with never-regress cases, scored per slice of intent and language.

Graders, cheapest first. Code checks facts, citations and behaviour. An LLM judge handles faithfulness and preference. Humans check the judge: accept it only if its kappa with people approaches the people's kappa with each other.

Judges are biased instruments. Position, verbosity and self-preference. Compare in both orders and count a win only if it survives the swap; judge with a different model; give a reference; use binary, checkable rubrics.

Faithful isn't correct. Score retrieval and generation separately. Smaller chunks dropped context recall from 0.91 to 0.78 while faithfulness rose: answers stuck to thinner context. The stale 30-day policy scored perfect faithfulness too.

Assume the model will be fooled. Injection arrives in seller policies, uploads and tool results, and no filter catches it reliably. So limit what a fooled model can do: narrow tools, server-side checks, confirmation for side effects.

Break the lethal trifecta. Private data plus untrusted content plus an outbound channel can leak data. The first two are the product, so remove the third: allow-listed links and images, email_return_label(order_id) instead of send_email.

• Shipping on a vibe check of 15 hand-picked chats.
• A classifier as the injection defence while the model picks amounts and recipients.
• A fallback model nobody ran against the golden set.

Day 5Cost, Caching and Operations

Four meters. Fresh input, cache write, cache read and output, each priced per model. History is resent every turn, so cost grows with the square of the turns. Compare models on cost per resolved conversation.

Prompt caching is cost and capacity. An exact prefix, built tools, system, then messages. A 5-minute write is 1.25× input, a read 0.1× or less, and for most Claude models reads don't count toward ITPM. Stable first, volatile last.

Semantic caches skip the guardrails. A false hit is a confident wrong answer that never met the model or Day 4's checks. Only general, non-personal questions; keys scoped by tenant, language and prompt version; purge from sources; measure false hits.

Route easy turns to a small model. A router decides before the call, a cascade after scoring the cheap answer. At $0.0012 vs $0.022 a conversation, a cascade pays until ~95% escalate. Keep each conversation on one model.

Budgets that degrade. Reserve the worst case from max_tokens, settle from usage, per feature, tenant, user and conversation. At 80%, thrifty; at 95%, minimal; spent, hand off. A step cap per turn stops loops first.

Fail over in steps. One retry layer, in the gateway, with SDK retries off. The fallback's cache is cold, so a conversation costs ~3× until it warms. Shift 10%, 25%, 50% to avoid its acceleration limits.

• A timestamp on line one of the system prompt, so every request misses the cache.
• Charting Anthropic's input_tokens alone, which excludes cached tokens.
• One organisation-wide budget, so a runaway loop takes the chat down.

BossDesign an AI Customer-Support Assistant

The model proposes, code decides. The model returns tool_use blocks; a deterministic controller runs the agent loop, checks every call in a tool gate, and decides. Ownership, amounts, limits, confirmation and idempotency live outside the model.

The safest argument doesn't exist. issue_refund takes only an order id: the gate computes the amount with refund_policy.allowed_amount(), over ₹5,000 hands off. email_return_label(order_id) lets the server pick the recipient. No free text leaves.

Proposals, confirmed and executed once. A side effect becomes a proposal row the customer confirms with a button showing the server's action. Confirming re-checks it and executes under an idempotency key from the proposal id, so double taps refund once.

Untrusted text only in tool results. Seller policies arrive inside lookup_order's result, JSON-encoded and labelled. The turn is then tainted, so any side effect needs confirmation. Where text sits lowers the odds; the gate decides safety.

A step cap, then instructive errors. Five tool calls per turn, then an is_error result telling the model to offer a human; hand off if it keeps going. Errors say what to do next, not just what failed.

Keep the tools array identical. For answers without actions, don't remove tools: they're the first cached segment, so every request would miss and hit 429s. The gate denies side effects instead, and pending proposals fail closed.

• issue_refund(order_id, amount) executed as soon as the model calls it.
• Removing tools from the request during an incident, breaking the cached prefix.
• Letting a continuation replay a side effect instead of the confirm endpoint.