Replication Trade-offs.

Yesterday, you learned how to split data across machines. Today, you learn how to copy it.

Replication seems simple: keep copies of data on multiple machines. But the moment you have copies, you face the hardest questions in distributed systems:

• Which copy is the "truth"?
• How long can copies be out of sync?
• What happens when the machine holding the truth dies?

By the end of this session, you won't just know what replication is—you'll understand exactly what you're trading with every replication decision, and you'll be able to design systems that handle these trade-offs explicitly.

Let's begin.

01Chapter 1: Why Replication Matters#

1.1 The Three Reasons to Replicate#

Every replicated system exists for one or more of these reasons:

Reason 1: High Availability#

If your database server dies, your application dies with it. With replicas, you can failover to another copy and keep serving requests.

The math: A single server with 99.9% uptime has about 8.8 hours of downtime per year. Two servers, either of which can serve requests, have combined uptime of 99.9999%—about 31 seconds of downtime per year (assuming independent failures).

Reason 2: Read Scaling#

A single database might handle 10,000 reads per second. With 5 read replicas, you can handle 50,000 reads per second. Writes still go to one place, but reads spread across all copies.

Reason 3: Geographic Distribution#

Users in Tokyo shouldn't wait for a round trip to a server in Virginia. With replicas in multiple regions, users read from nearby copies.

The physics: Light travels ~200km per millisecond in fiber. Tokyo to Virginia is ~11,000km = ~55ms minimum latency each way. A local replica reduces this to <10ms.

1.2 The Fundamental Trade-off#

Here's the truth about replication that every engineer must internalize:

You cannot have instantaneous, consistent replication without sacrificing availability or performance.

This isn't a limitation of current technology. It's physics and mathematics. The CAP theorem (and its extension, PACELC, which covers the latency trade-off when there is no partition) formalizes this, but the intuition is simple:

• Keeping replicas perfectly in sync requires coordination
• Coordination takes time (network round trips)
• During that time, you're either waiting (slower) or serving potentially stale data (inconsistent)

Every replication strategy is a different point on this trade-off spectrum. Your job is to choose the right point for your system.

02Chapter 2: Replication Architectures#

There are three fundamental architectures for replication. Each makes different trade-offs.

2.1 Leader-Follower (Primary-Replica)#

How It Works#

One node is the leader (also called primary or master). All writes go to the leader. The leader then sends changes to followers (also called replicas or secondaries). Reads can go to the leader or any follower.

%% title: Single-leader replication
%% steps:
%% W --> L: Every write goes to the one leader
%% L --> F1, L --> F2, L --> F3: The leader streams each change to all of its followers
%% RD --> F1, RD --> F2, RD --> F3: Reads are spread across the followers, so read capacity grows with each one
flowchart TD
  W@{ icon: "sd:users", label: "Writes" } --> L[("Leader")]:::core
  L -->|"Replication stream"| F1@{ icon: "sd:database", label: "Follower" }
  L -->|"Replication stream"| F2@{ icon: "sd:database", label: "Follower" }
  L -->|"Replication stream"| F3@{ icon: "sd:database", label: "Follower" }
  RD@{ icon: "sd:users", label: "Reads (distributed)" } --> F1
  RD --> F2
  RD --> F3
  class W,RD client
  class F1,F2,F3 store

Synchronous vs Asynchronous#

Synchronous replication: Leader waits for follower(s) to confirm the write before acknowledging to the client.

%% title: Synchronous replication
sequenceDiagram
  autonumber
  participant C as Client
  participant L as Leader
  participant F as Follower
  C->>L: Write
  L->>F: Replicate
  F-->>L: Confirm
  L-->>C: ACK

Asynchronous replication: Leader acknowledges to client immediately, replicates in the background.

%% title: Asynchronous replication
sequenceDiagram
  autonumber
  participant C as Client
  participant L as Leader
  participant F as Follower
  C->>L: Write
  L-->>C: ACK
  L-)F: replicate (background)

Semi-Synchronous: The Practical Middle Ground#

Many databases offer semi-synchronous replication (PostgreSQL and MySQL replicate asynchronously by default, and it's an option you turn on): wait for at least one follower to confirm, then acknowledge to client. Other followers replicate asynchronously.

%% title: Semi-synchronous replication
sequenceDiagram
  autonumber
  participant C as Client
  participant L as Leader
  participant F1 as Follower 1 (sync)
  participant F2 as Follower 2, 3, ...
  C->>L: Write
  L->>F1: Replicate
  F1-->>L: Confirm
  L-->>C: ACK
  L-)F2: replicate (async)

This gives you:

• Durability: At least one replica has every committed write
• Performance: Only wait for the fastest follower
• Availability: Can tolerate slow followers

When Leader-Follower Breaks#

Failure mode 1: Leader dies

With async replication, some writes may not have reached any follower. Those writes are lost.

With sync replication, a follower can be promoted to leader without data loss. But during failover (typically 10-60 seconds), writes fail.

Failure mode 2: Split brain

Network partition isolates the leader. Followers think it's dead and elect a new leader. Now you have two leaders accepting writes. When the network heals, you have conflicting data.

Failure mode 3: Replication lag

Under heavy load, followers fall behind. A read from a follower returns stale data. In extreme cases, followers can be minutes or hours behind.

2.2 Multi-Leader (Multi-Master)#

How It Works#

Multiple nodes accept writes. Each leader replicates to the others.

%% title: Multi-leader replication
flowchart TD
  W1@{ icon: "sd:users", label: "Writes" } --> L1[("Leader 1")]:::core
  W2@{ icon: "sd:users", label: "Writes" } --> L2[("Leader 2")]:::core
  L1 <-->|"Replication"| L2
  L1 --> F1@{ icon: "sd:database", label: "Follower" }
  L1 --> F2@{ icon: "sd:database", label: "Follower" }
  L2 --> F1
  L2 --> F2
  class W1,W2 client
  class F1,F2 store

Why Use Multi-Leader?#

The Conflict Problem#

When two leaders accept conflicting writes, what happens?

Scenario: User updates their email on their phone (hits EU leader) and their laptop (hits US leader) at the same time.

Both succeed locally. When they replicate to each other, which one wins?

Conflict resolution strategies:

1. Last-write-wins (LWW): Use timestamps; latest write wins. Simple but can lose data if clocks disagree.

2. Merge: For some data types, merge is possible. CRDTs (Conflict-free Replicated Data Types) are designed for this.

3. Application-level resolution: Notify the application of the conflict; let it decide.

4. Avoid conflicts: Design the system so conflicts are rare. For example, each user's data is primarily written in one region.

When Multi-Leader Breaks#

2.3 Leaderless (Dynamo-style)#

How It Works#

No designated leader. Clients write to multiple nodes directly. Reads also go to multiple nodes, and the client resolves differences.

%% title: Leaderless replication
flowchart TD
  W@{ icon: "sd:users", label: "Write" } --> N1@{ icon: "sd:database", label: "Node1" }
  W --> N2@{ icon: "sd:database", label: "Node2" }
  W --> N3@{ icon: "sd:database", label: "Node3" }
  N1 --- RQ@{ icon: "sd:users", label: "Read (quorum)" }
  N2 --- RQ
  N3 --- RQ
  class W,RQ client
  class N1,N2,N3 store

Quorums: The Math of Consistency#

With N replicas, you configure:

• W: Number of nodes that must confirm a write
• R: Number of nodes that must respond to a read

If W + R > N, the read and write sets overlap in at least one node, so a read normally sees the latest write. This is not an absolute guarantee: sloppy quorums, concurrent writes, and partially failed writes can still return stale values, and a quorum read is not necessarily linearizable.

Example with N=3:

• W=2, R=2: Read and write quorums overlap → reads normally see the latest write (strong-ish consistency)
• W=1, R=1: No overlap → eventual consistency, but faster

Sloppy Quorums and Hinted Handoff#

What if nodes are unavailable? Strict quorums would fail the request. Sloppy quorums allow writing to any available nodes, with hinted handoff—the temporary node holds the write and forwards it when the proper node recovers.

This improves availability but weakens consistency guarantees.

When Leaderless Breaks#

03Chapter 3: Replication Lag and Its Consequences#

3.1 What Is Replication Lag?#

In asynchronous replication, followers don't have the latest data immediately. The delay between a write on the leader and that write appearing on a follower is replication lag.

3.2 Anomalies Caused by Replication Lag#

Anomaly 1: Read Your Own Writes#

User submits a form. The write goes to the leader. User's next request reads from a follower that hasn't received the write yet. User sees the old data and thinks their submission failed.

Solutions:

• Read from leader for recently-written data
• Track write timestamps; only read from followers caught up to that point
• Sticky sessions: route user to the same follower consistently (only helps if that follower has caught up to the user's last write)

Anomaly 2: Monotonic Reads#

User makes two read requests. First hits follower A (caught up). Second hits follower B (lagging). User sees data go "backwards in time."

Solution: Sticky sessions or reading from the same replica within a session.

Anomaly 3: Consistent Prefix Reads#

Two causally related writes (A, then B) that land on different partitions may appear in a different order on a follower (within a single leader's log, order is preserved, so this shows up in partitioned databases).

Solution: Ensure causally related writes go through the same partition/leader, or use logical clocks.

3.3 Measuring and Monitoring Lag#

Every replicated system should expose:

1. Replication lag in seconds/bytes: How far behind is each follower?
2. Lag histogram: Distribution of lag across followers
3. Lag alerts: Page when lag exceeds acceptable threshold

-- PostgreSQL: Check replication lag
SELECT 
  client_addr,
  state,
  sent_lsn,
  write_lsn,
  flush_lsn,
  replay_lsn,
  pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes,
  replay_lag
FROM pg_stat_replication;

04Chapter 4: Consistency Models#

Different applications need different consistency guarantees. Choosing the wrong one means either poor performance (too strong) or confusing bugs (too weak).

4.1 Strong Consistency#

Every read returns the most recent write. All clients see the same data at the same time.

When you need it:

• Financial transactions
• Inventory counts (avoid overselling)
• Any case where stale reads cause real harm

4.2 Eventual Consistency#

If no new writes occur, all replicas will eventually converge to the same value. No guarantee about when.

When it's acceptable:

• Social media feeds (seeing a post 30 seconds late is fine)
• Analytics dashboards (approximate numbers are acceptable)
• Caches (by definition, allowed to be stale)

4.3 Read-Your-Writes Consistency#

A user always sees their own writes immediately. Other users may see stale data.

How to achieve it:

• Route user's reads to the leader for recently-written data
• Track user's last write timestamp; only read from caught-up replicas

When you need it:

• User profiles (user updates settings, expects to see them)
• Document editing (author's changes should appear immediately)
• Any self-referential read after write

4.4 Monotonic Reads#

Once a user sees a version of data, they never see an older version on subsequent reads.

How to achieve it: Sticky sessions (same replica per user session).

When you need it:

• Conversation threads (messages shouldn't disappear and reappear)
• Activity timelines (events shouldn't reorder)

4.5 Causal Consistency#

If event A causes event B, everyone sees A before B. Concurrent events may be seen in different orders.

How to achieve it: Logical clocks (vector clocks, Lamport timestamps) to track causality.

When you need it:

• Messaging (reply should appear after original message)
• Collaborative editing (operations that depend on each other must apply in order)

4.6 Choosing Your Consistency Level#

05Chapter 5: Extending the URL Shortener with Replication#

Yesterday, you designed a partitioned URL shortener. Today, we add replication for availability and read scaling.

5.1 Current State (From Day 1)#

%% title: Current state (from Day 1)
flowchart TD
  APP["App Servers"]:::core --> C@{ icon: "tech:redis", label: "Cache<br/>(Redis)" }
  C --> P0@{ icon: "sd:database", label: "Partition 0" }
  C --> P1@{ icon: "sd:database", label: "Partition 1" }
  C --> P2@{ icon: "sd:database", label: "Partition 2" }
  class C cache
  class P0,P1,P2 store

Each partition is a single PostgreSQL instance. If Partition 1 dies, all URLs mapped to it are unavailable.

5.2 Adding Read Replicas#

For each partition, we add read replicas:

%% title: Read replicas per partition
flowchart TD
  C@{ icon: "sd:cache", label: "Cache" }
  C --> P0[("Partition 0<br/>Primary")]:::core
  C --> P1[("Partition 1<br/>Primary")]:::core
  C --> P2[("Partition 2<br/>Primary")]:::core
  P0 --> P0R0@{ icon: "sd:database", label: "Replica 0" }
  P0 --> P0R1@{ icon: "sd:database", label: "Replica 1" }
  P1 --> P1R0@{ icon: "sd:database", label: "Replica 0" }
  P1 --> P1R1@{ icon: "sd:database", label: "Replica 1" }
  P2 --> P2R0@{ icon: "sd:database", label: "Replica 0" }
  P2 --> P2R1@{ icon: "sd:database", label: "Replica 1" }
  class C cache
  class P0R0,P0R1,P1R0,P1R1,P2R0,P2R1 store

5.3 Write Path#

Writes always go to the primary:

def create_short_url(short_code: str, original_url: str) -> URLMapping:
    partition = get_partition(short_code)
    primary = get_primary_connection(partition)
    
    result = primary.execute(
        "INSERT INTO urls (short_code, original_url, created_at) "
        "VALUES (%s, %s, NOW()) RETURNING created_at",
        (short_code, original_url)
    )
    
    return URLMapping(
        short_code=short_code,
        original_url=original_url,
        created_at=result.created_at
    )

5.4 Read Path#

Reads can go to any replica:

def redirect(short_code: str) -> Optional[str]:
    # Try cache first
    cached = cache.get(f"url:{short_code}")
    if cached:
        return cached
    
    # Cache miss - read from replica
    partition = get_partition(short_code)
    replica = get_random_replica(partition)  # Load balance across replicas
    
    result = replica.execute(
        "SELECT original_url FROM urls WHERE short_code = %s",
        (short_code,)
    )
    
    if result:
        cache.setex(f"url:{short_code}", 3600, result.original_url)
        return result.original_url
    
    return None

5.5 The Replication Lag Problem#

Scenario: User creates a short URL and immediately shares it.

The friend saw a 404 for a URL that exists. This is a read-your-writes violation—but for someone else's write.

5.6 Solutions for the Stale Read Problem#

Solution 1: Synchronous Replication#

Configure PostgreSQL for synchronous replication. Writes don't acknowledge until at least one replica confirms.

-- postgresql.conf on primary
-- 'on' only waits for the standby to flush WAL (durability);
-- 'remote_apply' waits until the change is visible to queries on the standby
synchronous_commit = remote_apply
synchronous_standby_names = 'replica1'

Trade-off: Every write is slower by one network round-trip to the replica (plus apply time with remote_apply). For a URL shortener doing 1,000 creates/second, adding 1ms per write might be acceptable. Note that only the synchronous standby is guaranteed fresh—reads that land on async replicas can still miss the URL.

Solution 2: Read from Primary for New URLs#

After creating a URL, read from primary for a short window:

def create_short_url(short_code: str, original_url: str) -> URLMapping:
    result = create_in_primary(short_code, original_url)
    
    # Mark this URL as "just created" for 5 seconds
    cache.setex(f"new:{short_code}", 5, "1")
    
    # Also cache the URL itself
    cache.setex(f"url:{short_code}", 3600, original_url)
    
    return result

def redirect(short_code: str) -> Optional[str]:
    cached = cache.get(f"url:{short_code}")
    if cached:
        return cached
    
    partition = get_partition(short_code)
    
    # Check if this is a newly created URL
    if cache.exists(f"new:{short_code}"):
        # Read from primary to ensure we see it
        db = get_primary_connection(partition)
    else:
        # Safe to read from replica
        db = get_random_replica(partition)
    
    result = db.execute(...)
    ...

Trade-off: Slightly more complex logic. Primary gets more reads for new URLs. But new URLs are also likely to be hot (just shared), so they'd probably end up cached quickly anyway.

Solution 3: Write-Through Cache#

Cache the URL at creation time, not at first read:

def create_short_url(short_code: str, original_url: str) -> URLMapping:
    result = create_in_primary(short_code, original_url)
    
    # Immediately cache - redirects will hit cache, not database
    cache.setex(f"url:{short_code}", 3600, original_url)
    
    return result

Trade-off: The cache is now in the write path. If Redis is slow or down, URL creation is affected. But for a URL shortener, this is probably acceptable since Redis is already critical infrastructure.

5.7 Failover Design#

What happens when a primary dies?

Automatic Failover#

Use a tool like Patroni (for PostgreSQL) or orchestration built into your database:

1. Primary stops responding to heartbeats
2. Followers detect primary is gone (timeout: 10-30 seconds)
3. Followers elect a new primary (consensus among remaining nodes)
4. Connection poolers (PgBouncer) are updated to point to new primary
5. Old primary, if it recovers, joins as a follower

Handling Writes During Failover#

During the failover window (10-60 seconds), writes fail. Options:

1. Return errors: Tell users "please retry." Simple but bad UX.

2. Queue writes: Accept writes into a queue, apply when new primary is elected. Complex, risk of inconsistency.

3. Fast failover: Optimize detection and promotion to minimize window. Best practical approach.

For a URL shortener, option 1 is probably fine. Users can retry creating a URL. It's not like a payment that might double-charge.

Handling Reads During Failover#

If the primary dies but replicas are healthy, reads continue to work (assuming you don't require reading from primary).

This is a major benefit of read replicas: read availability during primary failures.

06Chapter 6: Replication in Multi-Region Deployments#

6.1 The Latency Problem#

Your URL shortener is successful. Users are global. But all infrastructure is in us-east-1.

For redirects (the hot path), this latency is painful. Users expect near-instant redirects.

6.2 Read Replicas in Multiple Regions#

Deploy read replicas in each major region:

%% title: Regional read replicas
flowchart TD
  P[("us-east-1<br/>Primary Cluster")]:::core
  P --> EU@{ icon: "sd:database", label: "eu-west-1<br/>Read Replicas" }
  P --> AP@{ icon: "sd:database", label: "ap-northeast-1<br/>Read Replicas" }
  P --> SA@{ icon: "sd:database", label: "sa-east-1<br/>Read Replicas" }
  EU --- EUU@{ icon: "sd:users", label: "Europe" }
  AP --- APU@{ icon: "sd:users", label: "Tokyo" }
  SA --- SAU@{ icon: "sd:users", label: "São Paulo" }
  class EU,AP,SA store
  class EUU,APU,SAU client

Reads: Served from local region (~10ms) Writes: Go to us-east-1 (~80-150ms)

6.3 Dealing with Cross-Region Replication Lag#

Cross-region replication has higher lag than in-region (50-200ms typically).

The problem intensifies: User in Tokyo creates URL, shares with friend in Tokyo. Friend clicks. Local Tokyo replica doesn't have the URL yet.

Solutions:

1. Write-through to local cache: When URL is created, synchronously populate the Tokyo cache.

2. Route creator's reads to primary region: For a few seconds after creation, route that user's reads to us-east-1.

3. Accept the edge case: For a URL shortener, the friend can retry in 500ms. It's not ideal but might be acceptable.

6.4 Multi-Region Active-Active#

For write-heavy workloads or true global availability, you might want active-active:

%% title: Active-active regions
flowchart LR
  US[("us-east-1<br/>Primary<br/>(Americas)")]:::core <-->|"Bi-dir Repl."| EU[("eu-west-1<br/>Primary<br/>(Europe)")]:::core

Users in Americas write to us-east-1. Users in Europe write to eu-west-1. Changes replicate both directions.

The conflict problem: Same URL created in both regions simultaneously? With a URL shortener, this is unlikely (short codes are random), but you'd need conflict resolution if it happens.

Simpler alternative for URL shorteners: Keep single-primary, accept the write latency. URL creation is infrequent compared to redirects. A 150ms create is acceptable if redirects are 10ms.

07Chapter 7: Replication Internals#

Understanding how replication works under the hood helps you debug issues and make informed decisions.

7.1 Write-Ahead Log (WAL) Replication#

Most databases (PostgreSQL, MySQL, MongoDB) use log-based replication:

1. Every write is first recorded in a sequential log (WAL)
2. The log is shipped to replicas
3. Replicas apply the log entries to their data

%% title: WAL shipping
flowchart TD
  subgraph PRI["Primary"]
    direction TB
    W["Write"]:::client --> WAL[("WAL")]:::store
    WAL --> AD["Apply to data"]:::core
  end
  subgraph REP["Replica"]
    direction TB
    RW["Receive WAL"]:::service --> RA["Apply to data"]:::service
  end
  WAL -->|"Send to replicas"| RW

Why WAL?

• Sequential writes are fast (disk optimization)
• Log captures exact changes (no interpretation needed)
• Replicas can catch up by replaying log from any point

7.2 Logical vs Physical Replication#

7.3 Replication Slots and Catchup#

What if a replica goes offline for an hour?

-- Monitor replication slot disk usage
SELECT slot_name, 
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as retained_wal
FROM pg_replication_slots;

08Chapter 8: Handling Failover Correctly#

Failover sounds simple: promote a replica when the primary dies. In practice, it's full of edge cases.

8.1 The Split-Brain Problem#

Network partition isolates the primary:

%% title: Network partition isolates the primary
flowchart TD
  P[("Primary<br/>(isolated)")]:::bad x--x|"Network Partition · X X X"| R[("Replicas<br/>+ Application")]:::store
  P --- CA["Clients A"]:::client
  R --- CB["Clients B"]:::client

• Clients A continue writing to the old primary
• Replicas can't reach primary, promote a new one
• Clients B write to the new primary
• Network heals: two primaries with conflicting data

8.2 Fencing: The Solution to Split-Brain#

Fencing ensures the old primary can't accept writes after failover:

1. STONITH (Shoot The Other Node In The Head): Literally power off the old primary via IPMI/BMC.

2. Storage fencing: Revoke the old primary's access to shared storage.

3. Network fencing: Reconfigure network rules to block the old primary.

4. Lease-based: Primary must regularly renew a lease (in ZooKeeper/etcd). If it can't reach the lease store, it stops accepting writes.

# Lease-based fencing pseudocode
class Primary:
    def __init__(self):
        self.lease = self.acquire_lease()
    
    def accept_write(self, write):
        if not self.lease.is_valid():
            raise Exception("Lost leadership, rejecting write")
        
        self.execute(write)
    
    def background_loop(self):
        while True:
            try:
                self.lease.renew()
            except LeaseRenewalFailed:
                self.demote_to_read_only()
            sleep(lease_ttl / 3)

8.3 Failover Runbook#

A real failover runbook for our URL shortener:

09Chapter 9: The Hard Questions#

These are the questions you should be asking (and answering) in your design sessions.

9.1 "User creates URL, immediately shares it, friend clicks—what happens with async replication?"#

This is the core replication lag problem. Your answer should include:

1. Acknowledge the problem: Yes, with async replication, the friend might get a 404.

2. Quantify the risk: Replication lag is typically <100ms. The friend would have to click within 100ms of the share—unlikely but possible.

3. Propose mitigations:

   • Write-through cache at creation time (friend hits cache, not replica)
   • Semi-sync replication (wait for one replica before acknowledging)
   • Accept the edge case (friend retries, it works)

4. Make a recommendation: "For a URL shortener, I'd use write-through cache. It's simple, and we already have Redis in the architecture. The URL is cached before the user even finishes sharing."

9.2 "When would you accept stale reads? When not?"#

Accept stale reads:

• View counters, like counts (approximate is fine)
• News feeds (seeing a post 30 seconds late doesn't matter)
• Product listings (prices don't change every second)
• Analytics dashboards (real-time accuracy isn't required)

Reject stale reads:

• Account balances (stale balance → wrong financial decisions)
• Inventory counts (stale inventory → overselling)
• Permission checks (stale permissions → security hole)
• User authentication state (stale logout → security issue)

The URL shortener case:

• URL resolution: Can be eventually consistent. A URL created 1 second ago can tolerate <100ms lag.
• URL creation: Must be strongly consistent. User needs confirmation that the URL exists.
• URL deletion: Tricky. If we delete but replicas still serve it, is that a problem? For most URL shorteners, yes—the user deleted it for a reason.

9.3 "What replication factor would you use?"#

Replication factor = number of copies of each piece of data

Common choices:

• RF=1: No replication. Don't do this in production.
• RF=2: Can survive 1 node failure. Common for cost-sensitive workloads.
• RF=3: Can survive 1 failure with no degradation (quorum still possible). Standard choice.
• RF=5: Can survive 2 failures. For highly critical data.

For the URL shortener: RF=3 is appropriate. It's standard for production workloads, allows maintenance without risk, and the storage overhead is acceptable.

9.4 "How do you handle a replica that's fallen hours behind?"#

• Network issue: Fix network, let it catch up.
• Slow disk: Replica can't apply changes fast enough.
• Long-running query: Blocking replication on the replica.

Recovery options:

1. Let it catch up: If the primary is retaining WAL, the replica will eventually catch up.
2. Remove from rotation: Stop sending reads to this replica until it catches up.
3. Re-sync from scratch: If too far behind or WAL is gone, rebuild the replica from a backup.

For URL shortener: Remove from read rotation immediately. A replica hours behind is serving stale data. If catching up would take longer than a rebuild (or the WAL it needs is gone), rebuild it instead.

10Chapter 10: Session Summary#

What You Should Know Now#

After this session, you should be able to:

1. Explain leader-follower, multi-leader, and leaderless architectures with trade-offs
2. Design a replication strategy based on consistency and availability requirements
3. Identify replication lag anomalies and their solutions
4. Handle failover scenarios with explicit fencing strategies
5. Extend a partitioned system with replication (the URL shortener)

Key Trade-offs to Remember#

Questions to Ask in Every Design#

1. What consistency level does this data actually need?
2. What's the acceptable replication lag?
3. What happens during failover? How long is the outage?
4. Can readers tolerate stale data? For how long?
5. What's the strategy for a replica that falls far behind?

11Chapter 11: Real-World Interview Scenarios#

11.1 Conceptual Questions#

11.2 Design Questions#

%% title: Leader-follower with regional read replicas
flowchart TD
  P[("Primary (us-east-1)")]:::core --> EU[("EU Replicas")]:::store
  P --> AS[("Asia Replicas")]:::store
  P --> SA[("SA Replicas")]:::store

%% title: Sync to one replica, async to the rest
flowchart TD
  W["Write"]:::client --> P[("Primary")]:::core
  P --> S1[("Sync Replica 1")]:::store
  S1 --> ACK(["ACK"]):::ok
  P -.->|"for read scaling"| A[("Async Replica 2, 3")]:::store

11.3 Scenario-Based Questions#

11.4 Deep-Dive Questions#

12Chapter 12: Interview Preparation Checklist#

Before your interview, make sure you can:

Concepts#

Design Skills#

Operational Knowledge#

Real Systems#

Exercise 1: Consistency Level Selection#

For each scenario, choose the appropriate consistency level and justify:

1. Twitter-like feed: User posts a tweet, expects to see it immediately in their own feed
2. Inventory count: E-commerce site showing "Only 3 left in stock!"
3. Like counter: Showing "1.2M likes" on a viral post
4. Bank transfer: Moving money between accounts
5. Password change: User resets password, tries to log in immediately

Exercise 2: Failover Design#

Design the failover procedure for a three-node PostgreSQL cluster (1 primary, 2 replicas) running a payment processing system. Consider:

• Detection time
• Fencing mechanism
• Promotion criteria
• Application impact
• Data loss risk

Exercise 3: Replication Lag Mitigation#

Your application has a flow: user submits a form, you save to the database, then redirect to a "success" page that reads the data. Users are occasionally seeing "not found" errors. Design three different solutions with trade-offs for each.

Further Reading#

• Docs

  "Designing Data-Intensive Applications" Chapter 5: Replication (the definitive reference)

• Read

  Jepsen.io: Kyle Kingsbury's analyses of database consistency under failure

• Read

  GitHub's MySQL High Availability: How they handle replication and failover

• Read

  Stripe's Online Migrations: Maintaining consistency during schema changes

• Blog

  Patroni Documentation: Automatic failover for PostgreSQL

A.1 Replication Lag Monitoring#

from dataclasses import dataclass
from typing import List, Optional
import time

@dataclass
class ReplicaStatus:
    hostname: str
    lag_bytes: int
    lag_seconds: float
    state: str  # 'streaming', 'catchup', 'disconnected'
    
class ReplicationMonitor:
    def __init__(self, primary_conn, alert_threshold_seconds: float = 30):
        self.primary_conn = primary_conn
        self.alert_threshold = alert_threshold_seconds
    
    def get_replica_status(self) -> List[ReplicaStatus]:
        """Query primary for replication status."""
        query = """
            SELECT 
                client_addr as hostname,
                pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) as lag_bytes,
                COALESCE(EXTRACT(EPOCH FROM replay_lag), 0) as lag_seconds,
                state
            FROM pg_stat_replication
        """
        results = self.primary_conn.execute(query)
        
        return [
            ReplicaStatus(
                hostname=r['hostname'],
                lag_bytes=r['lag_bytes'],
                lag_seconds=r['lag_seconds'],
                state=r['state']
            )
            for r in results
        ]
    
    def check_health(self) -> List[str]:
        """Return list of alerts for unhealthy replicas."""
        alerts = []
        replicas = self.get_replica_status()
        
        for replica in replicas:
            if replica.state != 'streaming':
                alerts.append(f"{replica.hostname}: State is {replica.state}, not streaming")
            elif replica.lag_seconds > self.alert_threshold:
                alerts.append(f"{replica.hostname}: Lag is {replica.lag_seconds:.1f}s (threshold: {self.alert_threshold}s)")
        
        return alerts

A.2 Read-Your-Writes Consistency Helper#

import time
from typing import Optional
import redis

class ReadYourWritesRouter:
    """
    Routes reads to primary for recently-written keys,
    otherwise to replicas.
    """
    
    def __init__(
        self, 
        primary_conn,
        replica_pool,
        cache: redis.Redis,
        stickiness_seconds: float = 5.0
    ):
        self.primary = primary_conn
        self.replicas = replica_pool
        self.cache = cache
        self.stickiness_seconds = stickiness_seconds
    
    def record_write(self, key: str):
        """Call this after every write to track the key."""
        # Store timestamp of write
        self.cache.setex(
            f"written:{key}",
            int(self.stickiness_seconds),
            str(time.time())
        )
    
    def get_connection_for_read(self, key: str):
        """
        Returns primary connection if key was recently written,
        otherwise returns a replica connection.
        """
        written_at = self.cache.get(f"written:{key}")
        
        if written_at:
            # Recently written - use primary
            return self.primary
        else:
            # Safe to use replica
            return self.replicas.get_random()
    
    def read(self, key: str, query: str, params: tuple) -> Optional[dict]:
        """Execute a read with appropriate routing."""
        conn = self.get_connection_for_read(key)
        return conn.execute(query, params)
    
    def write(self, key: str, query: str, params: tuple) -> dict:
        """Execute a write and track it."""
        result = self.primary.execute(query, params)
        self.record_write(key)
        return result


# Usage example
router = ReadYourWritesRouter(primary, replica_pool, redis_client)

# Write - goes to primary, tracked for 5 seconds
router.write(
    key="user:123",
    query="UPDATE users SET email = %s WHERE id = %s",
    params=("new@example.com", 123)
)

# Immediate read - routed to primary because of recent write
result = router.read(
    key="user:123",
    query="SELECT * FROM users WHERE id = %s",
    params=(123,)
)

# Read after 5 seconds - safe to go to replica
time.sleep(5)
result = router.read(...)  # Goes to replica

A.3 Automatic Failover Detection#

import time
import threading
from enum import Enum
from typing import Callable, Optional

class NodeState(Enum):
    PRIMARY = "primary"
    REPLICA = "replica"
    FAILED = "failed"
    UNKNOWN = "unknown"

class FailoverDetector:
    """
    Monitors primary health and triggers failover callback
    when primary is determined to be down.
    """
    
    def __init__(
        self,
        primary_conn,
        health_check_interval: float = 1.0,
        failure_threshold: int = 3,
        on_failover: Optional[Callable] = None
    ):
        self.primary = primary_conn
        self.interval = health_check_interval
        self.threshold = failure_threshold
        self.on_failover = on_failover
        
        self.consecutive_failures = 0
        self.state = NodeState.UNKNOWN
        self._running = False
        self._thread = None
    
    def health_check(self) -> bool:
        """Returns True if primary is healthy."""
        try:
            result = self.primary.execute("SELECT 1", timeout=1.0)
            return result is not None
        except Exception:
            return False
    
    def _monitor_loop(self):
        while self._running:
            healthy = self.health_check()
            
            if healthy:
                self.consecutive_failures = 0
                self.state = NodeState.PRIMARY
            else:
                self.consecutive_failures += 1
                
                if self.consecutive_failures >= self.threshold:
                    self.state = NodeState.FAILED
                    
                    if self.on_failover:
                        self.on_failover()
                    
                    # Reset after triggering
                    self.consecutive_failures = 0
            
            time.sleep(self.interval)
    
    def start(self):
        self._running = True
        self._thread = threading.Thread(target=self._monitor_loop, daemon=True)
        self._thread.start()
    
    def stop(self):
        self._running = False
        if self._thread:
            self._thread.join()


# Usage
def handle_failover():
    print("Primary failed! Initiating failover...")
    # 1. Fence old primary
    # 2. Promote replica
    # 3. Update connection strings
    # 4. Alert on-call

detector = FailoverDetector(
    primary_conn=primary,
    health_check_interval=1.0,
    failure_threshold=3,
    on_failover=handle_failover
)

detector.start()