Preface#
Yesterday, we learned about workflow orchestration with Temporal — how to coordinate complex distributed transactions with durable execution. But there's a scenario that even the best orchestration can't handle:
| User opens shopping cart on phone (offline) | Simultaneously, user opens cart on laptop |
|---|---|
| Adds item A | Removes item A |
| Adds item B | Adds item D |
| Removes item C | Changes quantity of item C to 5 |
Both devices come online and sync.
Question: What should the cart contain?
- Option 1: Last-write-wins
- Phone synced last, so phone's version wins
- Lost: item D, the removal of A, the quantity change
- User is confused and frustrated
- Option 2: Laptop-write-wins
- Lost: item B, the phone's add of A, the removal of C
- User is confused and frustrated
- Option 3: Merge somehow?
- How do you merge "remove C" with "change C quantity to 5"?
- This is the hard problem.
This is the conflict resolution problem — when concurrent updates happen to the same data, how do we reconcile them?
Today, you'll learn:
- Why last-write-wins loses data
- Vector clocks for detecting conflicts
- CRDTs for automatic conflict resolution
- When to use each approach
A cart edited offline on a phone and a laptop, or two users incrementing 100 to 101, loses data under last-write-wins, which only suits values where the latest is best. Vector clocks detect when two writes are concurrent (A:1 vs C:1) instead of guessing. CRDTs merge automatically because merge is commutative, associative and idempotent: a G-Counter sums per-node counts, an OR-Set lets a concurrent add win, and each data type gets its own strategy.
Part I: Foundations#
Chapter 1: The Conflict Problem#
1.1 Why Conflicts Happen#
1. Network partitions
Node A (X = 1) ◄── PARTITION ──► Node B (X = 2)
Both nodes accept writes during partition. When partition heals: X = 1 or X = 2?
2. Concurrent clients
- Client 1: Read
X=0, WriteX=1(at T=0.001) - Client 2: Read
X=0, WriteX=2(at T=0.002)
Both read the same value, both write. Result depends on arrival order.
3. Offline operation
- Mobile app works offline.
- User makes changes.
- Syncs when online.
- Server has different changes from other devices.
4. Replication lag
- Write to primary:
X=1 - Async replication to replica (delayed)
- Read from replica:
X=0(stale) - Write based on stale read:
X=0+1=1(lost update)
1.2 The Lost Update Problem#
| Step | Counter |
|---|---|
| Initial | 100 |
| User A reads counter | 100 |
| User B reads counter | 100 |
| User A increments | writes 101 |
| User B increments | writes 101 |
Final value: 101 · Expected value: 102 — one increment was LOST.
This happens because:
- Both users read the same initial value
- Both computed new value independently
- Last write overwrote the first
In a distributed system
| Node 1 | Node 2 |
|---|---|
Cart: [A, B] |
Cart: [A, B] |
| User adds C | User adds D |
Cart: [A, B, C] |
Cart: [A, B, D] |
Then the nodes sync. Result with LWW:
- If Node 2 synced last:
[A, B, D] - Lost: item C
1.3 Conflict Resolution Strategies#
- 1. Last-write-wins (LWW)
- Simple: Timestamp decides winner
- Problem: Loses data
- Use when: Data loss is acceptable
- Example: Cache updates, status fields
- 2. First-write-wins
- First write is preserved
- Subsequent writes rejected
- Use when: Immutability matters
- Example: ID generation, audit logs
- 3. Application-level merge
- Application defines merge logic
- Most flexible, most complex
- Use when: Domain-specific rules exist
- Example: Git merge, document editing
- 4. User resolution
- Show conflict to user
- User chooses winner
- Use when: User input is available
- Example: File sync (Dropbox), contacts
- 5. Automatic merge (CRDTs)
- Data structures that auto-merge
- No conflicts by design
- Use when: Offline-first, collaboration
- Example: Shopping cart, collaborative text
Chapter 2: Last-Write-Wins (LWW)#
2.1 How LWW Works#
Each write includes a timestamp. On conflict, highest timestamp wins.
- Write 1:
{value: "A", timestamp: 1000} - Write 2:
{value: "B", timestamp: 1001}
Conflict resolution: "B" wins (higher timestamp)
Implementation — LWW register
State: {value: V, timestamp: T}
Write(new_value, new_timestamp):
if new_timestamp > state.timestamp:
state = {value: new_value, timestamp: new_timestamp}
Merge(other_state):
if other_state.timestamp > state.timestamp:
state = other_stateLWW for the picture and for the cart?
One store resolves every conflict by last-write-wins. It holds users' profile pictures and their shopping carts, and server clocks can drift apart by up to a second. What can go wrong for each, and for which one is LWW acceptable?
One strong answer
- Profile picture: if two changes race, one is dropped, and with skew the one that really came later can lose. The cost is a user seeing an older picture and setting it again. Acceptable: the latest value is the right one and nothing needs merging.
- Cart: concurrent adds from two devices are not alternatives; both matter. LWW keeps one cart and silently loses the other's items, and skew makes even the choice arbitrary. Not acceptable.
- Rule: LWW fits single values where latest is best, never collections or increments.
2.2 Problems with LWW#
- Clock skew
- Different machines have different clocks
- Write with "earlier" timestamp might actually be later
- NTP helps but doesn't guarantee synchronization
- Can lose the "actually latest" write
- Data loss
- Concurrent writes = one is discarded
- No merge, just overwrite
- Users lose work
- Acceptable only when losing a concurrent overwrite is harmless
- Tie-breaking
- What if timestamps are equal?
- Need secondary comparison (node ID, random)
- Still arbitrary, can lose data
- Millisecond precision often insufficient
- Causality violations
- LWW doesn't track causality
- Can't tell if writes are concurrent or sequential
- May overwrite with "older" logical state
- Confusing user experience
When LWW is acceptable
- Single-value updates (not increments)
- Idempotent operations
- Data can be regenerated
- User doesn't care about lost concurrent updates
Examples:
- User's current location (latest is best)
- Profile picture (latest is best)
- Cache entries (can re-fetch)
- "Last seen" timestamp
2.3 LWW Implementation#
# Last-Write-Wins Register Implementation
from dataclasses import dataclass
from datetime import datetime
from typing import Any, Optional, Tuple
import time
@dataclass
class LWWValue:
"""Value with timestamp for LWW."""
value: Any
timestamp: float # Unix timestamp with microseconds
node_id: str # For tie-breaking
def __lt__(self, other: 'LWWValue') -> bool:
"""Compare for ordering (used in conflict resolution)."""
if self.timestamp != other.timestamp:
return self.timestamp < other.timestamp
# Tie-break by node ID (deterministic)
return self.node_id < other.node_id
class LWWRegister:
"""
Last-Write-Wins Register.
Simple conflict resolution: latest timestamp wins.
"""
def __init__(self, node_id: str):
self.node_id = node_id
self.state: Optional[LWWValue] = None
def get(self) -> Optional[Any]:
"""Get current value."""
return self.state.value if self.state else None
def set(self, value: Any) -> LWWValue:
"""Set value with current timestamp."""
new_state = LWWValue(
value=value,
timestamp=time.time(),
node_id=self.node_id
)
if self.state is None or self.state < new_state:
self.state = new_state
return new_state
def merge(self, other: LWWValue):
"""Merge with value from another node."""
if self.state is None or self.state < other:
self.state = other
def get_state(self) -> Optional[LWWValue]:
"""Get full state for replication."""
return self.state
# Usage example
def demo_lww():
# Two nodes with same data
node_a = LWWRegister("node-a")
node_b = LWWRegister("node-b")
# Concurrent writes
state_a = node_a.set("value from A")
time.sleep(0.001) # Tiny delay
state_b = node_b.set("value from B")
# Sync
node_a.merge(state_b)
node_b.merge(state_a)
# Both nodes now have same value (B wins due to later timestamp)
print(node_a.get()) # "value from B"
print(node_b.get()) # "value from B"Chapter 3: Vector Clocks#
3.1 What Are Vector Clocks?#
Vector clocks track causality — they tell us whether one event happened before another, or if events are concurrent.
A vector clock is a map: {node_id: counter}. Each node maintains its own counter.
When a node does something:
- Increment its own counter
- Include full vector clock with message
When receiving a message:
- Merge vector clocks (take max of each counter)
- Increment own counter
Example
- Initial state — Node A:
{A: 0, B: 0, C: 0}· Node B:{A: 0, B: 0, C: 0}· Node C:{A: 0, B: 0, C: 0} - A writes X=1 — A's clock:
{A: 1, B: 0, C: 0} - A sends to B
- B receives
{A: 1, B: 0, C: 0} - B merges: max of each =
{A: 1, B: 0, C: 0} - B increments:
{A: 1, B: 1, C: 0}
- B receives
- Meanwhile, C writes X=2 (concurrent!) — C's clock:
{A: 0, B: 0, C: 1}
Now we can detect:
- A's write
{A: 1, B: 0, C: 0} - C's write
{A: 0, B: 0, C: 1}
Neither is "before" the other — they're CONCURRENT. We have a CONFLICT that needs resolution.
Before, after or concurrent?
Compare each pair of vector clocks. (a) {A:2, B:1, C:0} and {A:2, B:2, C:0}. (b) {A:3, B:0, C:1} and {A:2, B:1, C:1}. Which happened before which, and which pair is a conflict? What does the store do with each?
One strong answer
- (a) Before: every entry of the first is ≤ the second's, and B is strictly smaller. The second write supersedes the first; keep only the second.
- (b) Concurrent: the first is ahead on A, the second on B. Neither happened before the other, so it's a conflict: keep both versions (siblings) for a merge or the user to resolve.
- Why not compare totals or one entry? Happened-before needs every entry to be ≤; anything else hides a conflict.
3.2 Comparing Vector Clocks#
Given two vector clocks V1 and V2:
V1 < V2(V1 happened before V2):∀i: V1[i] ≤ V2[i]AND∃j: V1[j] < V2[j]V1 = V2(Same event):∀i: V1[i] = V2[i]V1 || V2(Concurrent): NeitherV1 < V2norV2 < V1(SomeV1[i] > V2[i]AND someV1[j] < V2[j])
Examples
{A:1, B:0} < {A:1, B:1}→ First happened before second{A:2, B:1} < {A:3, B:2}→ First happened before second{A:1, B:2} || {A:2, B:1}→ Concurrent (conflict!){A:1, B:1} = {A:1, B:1}→ Same
Why this matters
- If
V1 < V2: The write with V2 supersedes V1. No conflict — just apply V2. - If
V1 || V2: Concurrent writes — CONFLICT! Need resolution strategy.
Two replicas of cart [A, B] add C and D concurrently; last-write-wins keeps Node 2's [A, B, D] and silently loses C. Vector clocks detect the conflict: A writes {A:1, B:0, C:0}, B receives it and becomes {A:1, B:1, C:0}, C writes {A:0, B:0, C:1}; neither of A's and C's clocks is before the other, so the writes are concurrent.
3.3 Vector Clock Implementation#
# Vector Clock Implementation
from dataclasses import dataclass, field
from typing import Dict, Any, Optional, Tuple
from enum import Enum
class ClockComparison(Enum):
BEFORE = "before" # V1 < V2
AFTER = "after" # V1 > V2
EQUAL = "equal" # V1 = V2
CONCURRENT = "concurrent" # V1 || V2
@dataclass
class VectorClock:
"""
Vector clock for tracking causality.
"""
clocks: Dict[str, int] = field(default_factory=dict)
def increment(self, node_id: str) -> 'VectorClock':
"""Increment this node's clock."""
new_clocks = self.clocks.copy()
new_clocks[node_id] = new_clocks.get(node_id, 0) + 1
return VectorClock(clocks=new_clocks)
def merge(self, other: 'VectorClock') -> 'VectorClock':
"""Merge with another vector clock (take max of each)."""
all_nodes = set(self.clocks.keys()) | set(other.clocks.keys())
new_clocks = {
node: max(
self.clocks.get(node, 0),
other.clocks.get(node, 0)
)
for node in all_nodes
}
return VectorClock(clocks=new_clocks)
def compare(self, other: 'VectorClock') -> ClockComparison:
"""Compare with another vector clock."""
all_nodes = set(self.clocks.keys()) | set(other.clocks.keys())
self_greater = False
other_greater = False
for node in all_nodes:
self_val = self.clocks.get(node, 0)
other_val = other.clocks.get(node, 0)
if self_val > other_val:
self_greater = True
elif self_val < other_val:
other_greater = True
if self_greater and other_greater:
return ClockComparison.CONCURRENT
elif self_greater:
return ClockComparison.AFTER
elif other_greater:
return ClockComparison.BEFORE
else:
return ClockComparison.EQUAL
def copy(self) -> 'VectorClock':
"""Create a copy."""
return VectorClock(clocks=self.clocks.copy())
@dataclass
class VersionedValue:
"""Value with vector clock for conflict detection."""
value: Any
clock: VectorClock
class VectorClockStore:
"""
Key-value store with vector clock conflict detection.
"""
def __init__(self, node_id: str):
self.node_id = node_id
self.data: Dict[str, list[VersionedValue]] = {} # Key -> list of versions
def get(self, key: str) -> list[VersionedValue]:
"""
Get all versions of a key.
If there are conflicts, returns multiple versions.
Application must resolve.
"""
return self.data.get(key, [])
def put(
self,
key: str,
value: Any,
context: Optional[VectorClock] = None
) -> VectorClock:
"""
Put a value, optionally based on a previous read.
Args:
key: Key to write
value: New value
context: Vector clock from previous read (for conflict detection)
Returns:
New vector clock for this write
"""
# Get current versions
current_versions = self.data.get(key, [])
# Determine new clock
if context:
# Based on previous read - increment context
new_clock = context.increment(self.node_id)
else:
# Fresh write - merge all current clocks and increment
merged = VectorClock()
for version in current_versions:
merged = merged.merge(version.clock)
new_clock = merged.increment(self.node_id)
new_version = VersionedValue(value=value, clock=new_clock)
# Remove versions that this write supersedes
remaining = []
for version in current_versions:
comparison = new_clock.compare(version.clock)
if comparison == ClockComparison.AFTER:
# New write supersedes this version - discard
pass
elif comparison == ClockComparison.CONCURRENT:
# Conflict - keep both
remaining.append(version)
elif comparison == ClockComparison.BEFORE:
# This shouldn't happen if context was used correctly
remaining.append(version)
remaining.append(new_version)
self.data[key] = remaining
return new_clock
def merge_remote(self, key: str, remote_version: VersionedValue):
"""Merge a version received from another node."""
current_versions = self.data.get(key, [])
# Check if remote version supersedes, is superseded, or conflicts
remaining = []
dominated = False
for version in current_versions:
comparison = remote_version.clock.compare(version.clock)
if comparison == ClockComparison.AFTER:
# Remote supersedes local - discard local
pass
elif comparison == ClockComparison.BEFORE:
# Local supersedes remote - keep local, ignore remote
remaining.append(version)
dominated = True
elif comparison == ClockComparison.CONCURRENT:
# Conflict - keep both
remaining.append(version)
else: # EQUAL
# Same version - keep one
remaining.append(version)
dominated = True
if not dominated:
remaining.append(remote_version)
self.data[key] = remaining
# Example: Shopping cart with conflict detection
class ConflictAwareCart:
"""
Shopping cart that detects conflicts.
When conflicts exist, shows user both versions.
"""
def __init__(self, store: VectorClockStore, user_id: str):
self.store = store
self.user_id = user_id
self.last_read_clock: Optional[VectorClock] = None
def get_cart(self) -> Tuple[list[dict], bool]:
"""
Get cart contents.
Returns:
(items, has_conflicts)
"""
versions = self.store.get(f"cart:{self.user_id}")
if len(versions) == 0:
return [], False
if len(versions) == 1:
self.last_read_clock = versions[0].clock
return versions[0].value, False
# Multiple versions = conflict!
# For now, just return the first one and flag conflict
self.last_read_clock = versions[0].clock
return versions[0].value, True
def get_all_versions(self) -> list[list[dict]]:
"""Get all conflicting versions for user resolution."""
versions = self.store.get(f"cart:{self.user_id}")
return [v.value for v in versions]
def update_cart(self, items: list[dict]):
"""Update cart based on last read."""
self.last_read_clock = self.store.put(
f"cart:{self.user_id}",
items,
context=self.last_read_clock
)
def resolve_conflict(self, chosen_items: list[dict]):
"""Resolve conflict by choosing/merging versions."""
# Get all current versions
versions = self.store.get(f"cart:{self.user_id}")
# Merge all clocks
merged_clock = VectorClock()
for version in versions:
merged_clock = merged_clock.merge(version.clock)
# Write resolution with merged clock
self.last_read_clock = self.store.put(
f"cart:{self.user_id}",
chosen_items,
context=merged_clock
)Chapter 4: CRDTs (Conflict-free Replicated Data Types)#
4.1 What Are CRDTs?#
CRDTs are data structures that can be replicated across nodes, updated independently, and automatically merged without conflicts.
A CRDT guarantees:
- Any two replicas can be merged
- Merge is commutative:
merge(A, B) = merge(B, A) - Merge is associative:
merge(merge(A, B), C) = merge(A, merge(B, C)) - Merge is idempotent:
merge(A, A) = A
Result: Order of merges doesn't matter! All replicas eventually converge to same state.
Two types of CRDTs
| 1. State-based (CvRDT) | 2. Operation-based (CmRDT) |
|---|---|
| Ship entire state between nodes | Ship operations between nodes |
| Merge by combining states | Apply operations to reach same state |
| Simpler, but more bandwidth | Less bandwidth, but needs reliable (exactly-once, usually causal) delivery |
Design a counter that can't lose increments
Many nodes increment a like counter at the same time with no coordination, and they exchange their full state with each other at random, sometimes more than once. Design the state and the merge so every increment is counted exactly once, whatever order or how often states arrive. Then: how would you also allow decrements?
One strong answer
- State: one slot per node; each node only ever increments its own slot. Value = sum of all slots.
- Merge: per slot, take the max. A slot's larger value already includes the smaller one, so receiving the same state twice changes nothing (idempotent), and order doesn't matter (commutative, associative).
- Not sum on merge: re-delivered state would double-count.
- Decrements: keep a second grow-only counter for minus; value = plus − minus (a PN-Counter).
4.2 Common CRDT Types#
| # | CRDT | How it works | Use |
|---|---|---|---|
| 1 | G-Counter (Grow-only Counter) | Each node has its own counter · Can only increment · Value = sum of all node counters | View counts, likes |
| 2 | PN-Counter (Positive-Negative Counter) | Two G-Counters: one for + one for - · Value = positive - negative | Up/down votes, inventory |
| 3 | LWW-Register (Last-Writer-Wins Register) | Single value with timestamp · Highest timestamp wins | Simple fields, status |
| 4 | MV-Register (Multi-Value Register) | Keeps all concurrent values · Application resolves | When concurrent values matter |
| 5 | G-Set (Grow-only Set) | Can only add elements · Merge = union | Tags, followers |
| 6 | OR-Set (Observed-Remove Set) | Can add and remove elements · Add wins over concurrent remove | Shopping cart, todo list |
| 7 | LWW-Map (Last-Writer-Wins Map) | Map of key → LWW-Register | User profiles, settings |
| 8 | RGA (Replicated Growable Array) | Ordered list with insert/delete · Used for collaborative text | Google Docs-style editing |
4.3 CRDT Implementations#
# CRDT Implementations
from dataclasses import dataclass, field
from typing import Dict, Set, Any, Optional, Tuple
import time
import uuid
# =============================================================================
# G-Counter (Grow-only Counter)
# =============================================================================
class GCounter:
"""
Grow-only counter CRDT.
Each node can only increment its own counter.
Value is sum of all counters.
"""
def __init__(self, node_id: str):
self.node_id = node_id
self.counts: Dict[str, int] = {}
def increment(self, amount: int = 1):
"""Increment this node's counter."""
if amount < 0:
raise ValueError("G-Counter can only increment")
self.counts[self.node_id] = self.counts.get(self.node_id, 0) + amount
def value(self) -> int:
"""Get total count across all nodes."""
return sum(self.counts.values())
def merge(self, other: 'GCounter'):
"""Merge with another G-Counter."""
all_nodes = set(self.counts.keys()) | set(other.counts.keys())
self.counts = {
node: max(
self.counts.get(node, 0),
other.counts.get(node, 0)
)
for node in all_nodes
}
def get_state(self) -> Dict[str, int]:
"""Get state for replication."""
return self.counts.copy()
@classmethod
def from_state(cls, node_id: str, state: Dict[str, int]) -> 'GCounter':
"""Create from replicated state."""
counter = cls(node_id)
counter.counts = state.copy()
return counter
# =============================================================================
# PN-Counter (Positive-Negative Counter)
# =============================================================================
class PNCounter:
"""
Positive-Negative counter CRDT.
Supports both increment and decrement.
Uses two G-Counters internally.
"""
def __init__(self, node_id: str):
self.positive = GCounter(node_id)
self.negative = GCounter(node_id)
def increment(self, amount: int = 1):
"""Increment counter."""
self.positive.increment(amount)
def decrement(self, amount: int = 1):
"""Decrement counter."""
self.negative.increment(amount)
def value(self) -> int:
"""Get current value (positive - negative)."""
return self.positive.value() - self.negative.value()
def merge(self, other: 'PNCounter'):
"""Merge with another PN-Counter."""
self.positive.merge(other.positive)
self.negative.merge(other.negative)
# =============================================================================
# G-Set (Grow-only Set)
# =============================================================================
class GSet:
"""
Grow-only set CRDT.
Elements can only be added, never removed.
Merge is union.
"""
def __init__(self):
self.elements: Set[Any] = set()
def add(self, element: Any):
"""Add element to set."""
self.elements.add(element)
def contains(self, element: Any) -> bool:
"""Check if element is in set."""
return element in self.elements
def value(self) -> Set[Any]:
"""Get all elements."""
return self.elements.copy()
def merge(self, other: 'GSet'):
"""Merge with another G-Set (union)."""
self.elements = self.elements | other.elements
# =============================================================================
# OR-Set (Observed-Remove Set) - Add-Wins
# =============================================================================
@dataclass
class ORSetElement:
"""Element in OR-Set with unique tags."""
value: Any
tags: Set[str] = field(default_factory=set)
class ORSet:
"""
Observed-Remove Set CRDT.
Supports both add and remove.
Add wins over concurrent remove.
Each add creates a unique tag.
Remove deletes all known tags.
If any tag survives, element is present.
"""
def __init__(self, node_id: str):
self.node_id = node_id
# Map from value to set of (tag, tombstoned) pairs
self.elements: Dict[Any, Dict[str, bool]] = {}
def add(self, value: Any):
"""Add element with new unique tag."""
tag = f"{self.node_id}:{uuid.uuid4()}"
if value not in self.elements:
self.elements[value] = {}
self.elements[value][tag] = False # Not tombstoned
def remove(self, value: Any):
"""Remove element by tombstoning all known tags."""
if value in self.elements:
for tag in self.elements[value]:
self.elements[value][tag] = True # Tombstone
def contains(self, value: Any) -> bool:
"""Check if element is in set (any non-tombstoned tag)."""
if value not in self.elements:
return False
return any(
not tombstoned
for tombstoned in self.elements[value].values()
)
def value(self) -> Set[Any]:
"""Get all non-removed elements."""
return {
v for v in self.elements
if self.contains(v)
}
def merge(self, other: 'ORSet'):
"""Merge with another OR-Set."""
all_values = set(self.elements.keys()) | set(other.elements.keys())
for value in all_values:
self_tags = self.elements.get(value, {})
other_tags = other.elements.get(value, {})
# Merge tags
merged_tags = {}
all_tag_ids = set(self_tags.keys()) | set(other_tags.keys())
for tag in all_tag_ids:
# Tag is tombstoned if tombstoned in either
self_tomb = self_tags.get(tag, True) # Missing = already removed
other_tomb = other_tags.get(tag, True)
# If tag exists in one but not other:
# - If not tombstoned in the one that has it, keep it
# - If tombstoned in the one that has it, it's removed
if tag in self_tags and tag in other_tags:
merged_tags[tag] = self_tomb or other_tomb
elif tag in self_tags:
merged_tags[tag] = self_tomb
else:
merged_tags[tag] = other_tomb
if merged_tags:
self.elements[value] = merged_tags
# =============================================================================
# LWW-Map (Last-Writer-Wins Map)
# =============================================================================
@dataclass
class LWWEntry:
"""Entry in LWW-Map."""
value: Any
timestamp: float
deleted: bool = False
class LWWMap:
"""
Last-Writer-Wins Map CRDT.
Each key is a LWW-Register.
Supports add, update, and delete.
"""
def __init__(self, node_id: str):
self.node_id = node_id
self.entries: Dict[str, LWWEntry] = {}
def set(self, key: str, value: Any):
"""Set a key-value pair."""
timestamp = time.time()
current = self.entries.get(key)
if current is None or current.timestamp < timestamp:
self.entries[key] = LWWEntry(
value=value,
timestamp=timestamp,
deleted=False
)
def delete(self, key: str):
"""Delete a key."""
timestamp = time.time()
current = self.entries.get(key)
if current is None or current.timestamp < timestamp:
self.entries[key] = LWWEntry(
value=None,
timestamp=timestamp,
deleted=True
)
def get(self, key: str) -> Optional[Any]:
"""Get value for key."""
entry = self.entries.get(key)
if entry is None or entry.deleted:
return None
return entry.value
def keys(self) -> Set[str]:
"""Get all non-deleted keys."""
return {
k for k, v in self.entries.items()
if not v.deleted
}
def to_dict(self) -> Dict[str, Any]:
"""Get as regular dict."""
return {
k: v.value
for k, v in self.entries.items()
if not v.deleted
}
def merge(self, other: 'LWWMap'):
"""Merge with another LWW-Map."""
all_keys = set(self.entries.keys()) | set(other.entries.keys())
for key in all_keys:
self_entry = self.entries.get(key)
other_entry = other.entries.get(key)
if self_entry is None:
self.entries[key] = other_entry
elif other_entry is None:
pass # Keep self
elif other_entry.timestamp > self_entry.timestamp:
self.entries[key] = other_entry
# If timestamps equal, keep self (arbitrary but deterministic)
# =============================================================================
# Shopping Cart CRDT (OR-Set based)
# =============================================================================
@dataclass
class CartItem:
"""Item in shopping cart."""
product_id: str
quantity: int
def __hash__(self):
return hash(self.product_id)
def __eq__(self, other):
if isinstance(other, CartItem):
return self.product_id == other.product_id
return False
class ShoppingCartCRDT:
"""
Shopping cart using OR-Set + PN-Counter.
OR-Set for items (add/remove)
PN-Counter for quantities
"""
def __init__(self, node_id: str):
self.node_id = node_id
self.items = ORSet(node_id) # Tracks which items are in cart
self.quantities: Dict[str, PNCounter] = {} # Quantity per item
def add_item(self, product_id: str, quantity: int = 1):
"""Add item to cart."""
self.items.add(product_id)
if product_id not in self.quantities:
self.quantities[product_id] = PNCounter(self.node_id)
self.quantities[product_id].increment(quantity)
def remove_item(self, product_id: str):
"""Remove item from cart entirely."""
self.items.remove(product_id)
if product_id in self.quantities:
# Cancel the observed quantity so a later re-add starts from zero
observed = self.quantities[product_id].value()
if observed > 0:
self.quantities[product_id].decrement(observed)
def update_quantity(self, product_id: str, delta: int):
"""Change quantity (positive or negative)."""
if product_id not in self.quantities:
self.quantities[product_id] = PNCounter(self.node_id)
if delta > 0:
self.quantities[product_id].increment(delta)
else:
self.quantities[product_id].decrement(abs(delta))
def get_items(self) -> Dict[str, int]:
"""Get cart contents."""
result = {}
for product_id in self.items.value():
if product_id in self.quantities:
qty = self.quantities[product_id].value()
if qty > 0:
result[product_id] = qty
return result
def merge(self, other: 'ShoppingCartCRDT'):
"""Merge with another cart."""
# Merge item sets
self.items.merge(other.items)
# Merge quantity counters
all_products = set(self.quantities.keys()) | set(other.quantities.keys())
for product_id in all_products:
if product_id not in self.quantities:
self.quantities[product_id] = PNCounter(self.node_id)
if product_id in other.quantities:
self.quantities[product_id].merge(other.quantities[product_id])
# Usage example
def demo_cart_crdt():
# Two devices, same user
phone = ShoppingCartCRDT("phone")
laptop = ShoppingCartCRDT("laptop")
# Phone adds items
phone.add_item("apple", 3)
phone.add_item("banana", 2)
# Laptop adds different items (concurrently, offline)
laptop.add_item("orange", 5)
laptop.add_item("apple", 1) # Also adds apple!
# Sync
phone.merge(laptop)
laptop.merge(phone)
# Both devices have same cart
print(phone.get_items())
# {'apple': 4, 'banana': 2, 'orange': 5}
# Apple quantity = 3 + 1 = 4 (merged!)
print(laptop.get_items())
# {'apple': 4, 'banana': 2, 'orange': 5}LWW, vector clocks and CRDTs
Q1/4 A key has two versions with vector clocks
{A:3, B:1}and{A:2, B:4}. What should the store do?Answer & explanation
Answer: A
One clock is ahead on A and the other is ahead on B, so neither happened before the other: that is a conflict. Summing counters or comparing a single entry is meaningless; happened-before needs every entry to be less than or equal.
Q2/4 Node P's clock runs 2 seconds fast. A user sets their status via P, then a second later changes it again via node Q. Under last-write-wins, which value survives?
Answer & explanation
Answer: C
LWW trusts timestamps, so clock skew can make the actually-latest write lose and be silently discarded. NTP narrows skew but does not guarantee it, and LWW never keeps both values.
Q3/4 Offline, the phone removes item C from the cart while the laptop adds C again. Both carts are OR-Sets. After they sync, is C in the cart?
Answer & explanation
Answer: A
An OR-Set remove only tombstones the tags it has observed. A concurrent add carries a fresh tag that survives the merge. "Which synced last" is LWW thinking; OR-Set merges are order-independent by design.
Q4/4 Why does a G-Counter merge take the max of each node's count instead of adding them?
Answer & explanation
Answer: B
Each node only increments its own slot, so the larger value per node already includes the smaller one. Taking the max makes
merge(A, A) = A. The total is still a sum across nodes; it is the per-node merge that uses max.
Part II: Production Implementation#
Chapter 5: Choosing the Right Approach#
# Decision Framework for Conflict Resolution
from enum import Enum
from dataclasses import dataclass
from typing import Any, Callable, Optional
class ConflictStrategy(Enum):
LAST_WRITE_WINS = "lww"
VECTOR_CLOCK = "vector_clock"
CRDT = "crdt"
APPLICATION_MERGE = "app_merge"
USER_RESOLUTION = "user_resolution"
@dataclass
class DataCharacteristics:
"""Characteristics of data to determine conflict strategy."""
# Can data be regenerated if lost?
regenerable: bool
# Is latest value always the "correct" value?
latest_is_correct: bool
# Can concurrent updates be automatically merged?
mergeable: bool
# Is user available to resolve conflicts?
user_available: bool
# Is data structure simple (register) or complex (set, counter)?
is_simple_value: bool
# Does order of operations matter?
order_dependent: bool
def choose_strategy(characteristics: DataCharacteristics) -> ConflictStrategy:
"""
Choose conflict resolution strategy based on data characteristics.
"""
# If latest is correct and data is simple, LWW is fine
if characteristics.latest_is_correct and characteristics.is_simple_value:
return ConflictStrategy.LAST_WRITE_WINS
# If data is naturally mergeable (counters, sets), use CRDT
if characteristics.mergeable and not characteristics.order_dependent:
return ConflictStrategy.CRDT
# If user is available and data is important, ask them
if characteristics.user_available and not characteristics.regenerable:
return ConflictStrategy.USER_RESOLUTION
# If order matters or complex merge logic needed
if characteristics.order_dependent:
return ConflictStrategy.VECTOR_CLOCK
# Default to application-level merge
return ConflictStrategy.APPLICATION_MERGE
# Example usage
STRATEGY_MAPPING = {
"user_location": choose_strategy(DataCharacteristics(
regenerable=True,
latest_is_correct=True,
mergeable=False,
user_available=False,
is_simple_value=True,
order_dependent=False
)), # LWW
"shopping_cart": choose_strategy(DataCharacteristics(
regenerable=False,
latest_is_correct=False,
mergeable=True,
user_available=False,
is_simple_value=False,
order_dependent=False
)), # CRDT
"document": choose_strategy(DataCharacteristics(
regenerable=False,
latest_is_correct=False,
mergeable=True,
user_available=True,
is_simple_value=False,
order_dependent=True
)), # VECTOR_CLOCK or USER_RESOLUTION
"view_count": choose_strategy(DataCharacteristics(
regenerable=True,
latest_is_correct=False,
mergeable=True,
user_available=False,
is_simple_value=True,
order_dependent=False
)), # CRDT (G-Counter)
}Chapter 6: Real-World CRDT Systems#
# Production CRDT System
import asyncio
from dataclasses import dataclass, field
from typing import Dict, Any, Set, Optional, List
from datetime import datetime, timezone
import json
@dataclass
class ReplicationMessage:
"""Message for CRDT state replication."""
source_node: str
timestamp: datetime
crdt_type: str
key: str
state: Any
class CRDTReplicationManager:
"""
Manages CRDT replication across nodes.
Handles:
- State-based replication
- Gossip protocol
- Anti-entropy
"""
def __init__(
self,
node_id: str,
peer_nodes: List[str],
message_queue, # Kafka, RabbitMQ, etc.
):
self.node_id = node_id
self.peer_nodes = peer_nodes
self.mq = message_queue
# Local CRDT stores
self.counters: Dict[str, PNCounter] = {}
self.sets: Dict[str, ORSet] = {}
self.maps: Dict[str, LWWMap] = {}
# Track what we've sent to avoid redundant replication
self.last_sent: Dict[str, datetime] = {}
async def start(self):
"""Start replication manager."""
# Start background tasks
asyncio.create_task(self._receive_loop())
asyncio.create_task(self._anti_entropy_loop())
# =========================================================================
# Local Operations
# =========================================================================
def get_counter(self, key: str) -> PNCounter:
"""Get or create counter."""
if key not in self.counters:
self.counters[key] = PNCounter(self.node_id)
return self.counters[key]
def increment_counter(self, key: str, amount: int = 1):
"""Increment counter and trigger replication."""
counter = self.get_counter(key)
counter.increment(amount)
asyncio.create_task(self._replicate_counter(key))
def get_set(self, key: str) -> ORSet:
"""Get or create OR-Set."""
if key not in self.sets:
self.sets[key] = ORSet(self.node_id)
return self.sets[key]
def add_to_set(self, key: str, value: Any):
"""Add to set and trigger replication."""
s = self.get_set(key)
s.add(value)
asyncio.create_task(self._replicate_set(key))
def remove_from_set(self, key: str, value: Any):
"""Remove from set and trigger replication."""
s = self.get_set(key)
s.remove(value)
asyncio.create_task(self._replicate_set(key))
# =========================================================================
# Replication
# =========================================================================
async def _replicate_counter(self, key: str):
"""Replicate counter state to peers."""
counter = self.counters[key]
message = ReplicationMessage(
source_node=self.node_id,
timestamp=datetime.now(timezone.utc),
crdt_type="pn_counter",
key=key,
state={
"positive": counter.positive.get_state(),
"negative": counter.negative.get_state()
}
)
await self.mq.publish("crdt_replication", json.dumps(message.__dict__, default=str))
async def _replicate_set(self, key: str):
"""Replicate set state to peers."""
s = self.sets[key]
message = ReplicationMessage(
source_node=self.node_id,
timestamp=datetime.now(timezone.utc),
crdt_type="or_set",
key=key,
state=self._serialize_or_set(s)
)
await self.mq.publish("crdt_replication", json.dumps(message.__dict__, default=str))
async def _receive_loop(self):
"""Receive and apply replication messages."""
async for raw_message in self.mq.subscribe("crdt_replication"):
try:
data = json.loads(raw_message)
# Skip our own messages
if data["source_node"] == self.node_id:
continue
await self._apply_replication(data)
except Exception as e:
logger.error(f"Failed to apply replication: {e}")
async def _apply_replication(self, data: dict):
"""Apply a replication message."""
crdt_type = data["crdt_type"]
key = data["key"]
state = data["state"]
if crdt_type == "pn_counter":
counter = self.get_counter(key)
# Create remote counter from state
remote = PNCounter(data["source_node"])
remote.positive.counts = state["positive"]
remote.negative.counts = state["negative"]
# Merge
counter.merge(remote)
elif crdt_type == "or_set":
s = self.get_set(key)
remote = self._deserialize_or_set(state, data["source_node"])
s.merge(remote)
async def _anti_entropy_loop(self):
"""
Periodically sync full state with random peer.
Catches any missed updates.
"""
while True:
await asyncio.sleep(60) # Every minute
# Pick random peer
import random
peer = random.choice(self.peer_nodes)
# Request full state from peer
await self._sync_with_peer(peer)
async def _sync_with_peer(self, peer: str):
"""Full state sync with a peer."""
# In practice, this would be a direct RPC call
# For now, just broadcast our full state
for key, counter in self.counters.items():
await self._replicate_counter(key)
for key, s in self.sets.items():
await self._replicate_set(key)
def _serialize_or_set(self, s: ORSet) -> dict:
"""Serialize OR-Set for transmission."""
return {
str(k): {tag: tomb for tag, tomb in v.items()}
for k, v in s.elements.items()
}
def _deserialize_or_set(self, data: dict, node_id: str) -> ORSet:
"""Deserialize OR-Set from transmission."""
s = ORSet(node_id)
for k, tags in data.items():
s.elements[k] = {tag: tomb for tag, tomb in tags.items()}
return sChoosing and replicating CRDTs
Q1/3 Using the lesson's decision framework, which data is the best fit for last-write-wins?
Answer & explanation
Answer: B
When the latest value is the correct one and the data is a simple value, LWW loses nothing that matters. Carts and view counts have concurrent updates that should all count, so they map to CRDTs; documents need richer merging.
Q2/3 The replication manager already publishes state on every change. Why does it also run an anti-entropy sync every minute?
Answer & explanation
Answer: A
Messages can be dropped or a node can be offline; periodic full-state sync repairs those gaps. CRDT merges have no unresolvable conflicts, and there is no authoritative node; every replica converges.
Q3/3 The system replicates whole PN-Counter state instead of "increment by 1" operation messages. What does that buy?
Answer & explanation
Answer: D
State-based CRDTs cost more bandwidth but tolerate redelivery and reordering, because merging the same state again changes nothing. Operation-based CRDTs save bandwidth but depend on reliable delivery; a duplicated "+1" would be counted twice.
Part III: Real-World Application#
Chapter 7: Case Studies#
7.1 Case Study: Riak (Dynamo-style)#
Riak is a distributed key-value store using:
- Vector clocks for conflict detection
- Sibling values for concurrent updates
- Application-level or automatic resolution
How it works
- Write with context (vector clock from read)
- If context matches, write succeeds
- If concurrent write detected, create "sibling"
- On read, return all siblings for resolution
Configuration options
allow_mult: true→ Keep siblings, application resolvesallow_mult: false→ LWW, last write wins
Built-in CRDT support
- Counters (PN-Counter)
- Sets (OR-Set)
- Maps (nested CRDTs)
- Flags (boolean, only inside maps)
- Registers (LWW, only inside maps)
Example: Riak Set
# Add to set
riak.update_set("bucket", "shopping-cart",
add=["item-1", "item-2"])
# Remove from set
riak.update_set("bucket", "shopping-cart",
remove=["item-1"])
# Automatic merge across replicas7.2 Case Study: Google Docs (OT/CRDT)#
Google Docs uses Operational Transformation (OT), but CRDTs are becoming more common (Yjs, Automerge).
Operational transformation
- Client A: Insert "X" at position 5
- Client B: Insert "Y" at position 3
Without transformation (document starts as "abcdefghij"):
- A already applied its own X (
"abcdeXfghij"), then B's op: Insert "Y" at 3 →"abcYdeXfghij" - B already applied its own Y (
"abcYdefghij"), then A's op: Insert "X" at 5 →"abcYdXefghij" - Result: Different documents!
With transformation:
- At B: A's insert at 5 is transformed against B's earlier insert at 3 (which shifted the text) → Insert "X" at position 6
- At A: B's insert at 3 is before position 5, so it stays: Insert "Y" at position 3
- Both get:
"abcYdeXfghij"
CRDT alternative (RGA - Replicated Growable Array)
- Each character has unique ID:
(node_id, sequence) - Insert references the character it comes after.
- Deletes are tombstones.
Advantages:
- No central server needed
- Works offline
- Guaranteed convergence
- Simpler than OT
Related sequence CRDTs in use: Automerge (RGA-based), Yjs (YATA algorithm)
7.3 Case Study: Redis CRDTs#
Redis Enterprise supports CRDTs for active-active replication.
Supported types
- CRDT Strings (LWW) —
SET key value→ Last write wins across datacenters - CRDT Counters —
INCRBY key amount→ Merged across datacenters (no lost increments) - CRDT Sets —
SADD key member,SREM key member→ Add-wins semantics - CRDT Sorted Sets —
ZADD key score member→ add-wins for membership; score treated as a counter (ZINCRBYsums, concurrentZADDof the same member resolves LWW) - CRDT Hashes —
HSET key field value→ LWW per field
Conflict resolution rules
| Type | Rule |
|---|---|
| Strings | LWW based on wall-clock + vector clock |
| Counters | Sum of all increments |
| Sets | Union (add-wins) |
| Sorted Sets | Add-wins (observed-remove) for membership; score resolved like a counter (ZINCRBY sums, concurrent ZADD is LWW) |
| Hashes | LWW per field |
Example: Global Session Store
# Datacenter US
HSET session:user123 last_seen "2024-01-15T10:00:00Z"
HSET session:user123 cart_count 5
# Datacenter EU (concurrent)
HSET session:user123 last_seen "2024-01-15T10:00:01Z"
HSET session:user123 preferences "dark_mode"
# After sync, both have:
{
last_seen: "2024-01-15T10:00:01Z", # LWW
cart_count: 5,
preferences: "dark_mode"
}Chapter 8: Common Mistakes#
8.1 Mistake 1: Using LWW When Order Matters#
WRONG: LWW for sequential operations
# Counter implemented with LWW
counter_value = 0
# User A reads: 0
# User B reads: 0
# User A writes: 1 (timestamp: 1000)
# User B writes: 1 (timestamp: 1001)
# Result: 1 (should be 2!)CORRECT: Use CRDT counter
counter = PNCounter("node-1")
# User A increments
counter.increment()
# User B increments (concurrent)
other_counter = PNCounter("node-2")
other_counter.increment()
# Merge
counter.merge(other_counter)
# Result: 2 (ok)8.2 Mistake 2: Not Handling Concurrent Removes#
WRONG: Simple set with concurrent add/remove
# Initial: {A, B}
# Node 1: Remove B
set1 = {A}
# Node 2 (concurrent): Add C
set2 = {A, B, C}
# Merge with union: {A, B, C}
# But Node 1 wanted B removed!CORRECT: Use OR-Set
# Initial: {A, B}
orset = ORSet("node-1")
orset.add("A")
orset.add("B")
# Node 2 syncs the initial state
orset2 = ORSet("node-2")
orset2.merge(orset) # Has A, B
# Node 1: Remove B (tombstones B's tags)
orset.remove("B")
# Node 2 (concurrent): Add C (new tag)
orset2.add("C")
# Merge
orset.merge(orset2)
# Result: {A, C}
# B's tags were tombstoned, C has new tagHow big is a per-device vector clock?
A notes app gives every device its own vector-clock entry, and a popular shared key has been written from a million devices. Assume an entry (device ID plus counter) takes about 20 bytes. How big is that key's clock, and what does every write and every comparison have to handle? What if clients wrote through three server nodes instead?
One strong answer
- Per-device: 1,000,000 × 20 B ≈ 20 MB of clock for one key, shipped with every replicated write and walked entry by entry on every comparison.
- Per-server: 3 entries × 20 B ≈ 60 bytes, fixed no matter how many devices write.
- So clock size grows with the number of writers it tracks: make the servers the writers.
8.3 Mistake 3: Vector Clock Explosion#
WRONG: Creating new vector clock entry per client
# Every client gets its own entry
{
"client-1": 1,
"client-2": 1,
"client-3": 1,
...
"client-1000000": 1
}
# Vector clock becomes huge!
# Memory explosion, slow comparisonsCORRECT: Use node-based clocks, not client-based
# Clients write through servers
# Vector clock has entry per SERVER, not per CLIENT
{
"server-us-east": 1000,
"server-us-west": 999,
"server-eu": 1001
}
# Fixed number of entries (number of servers)8.4 Mistake Checklist#
- Don't use LWW for additive operations — Use counters, sets
- Don't use simple sets — Use OR-Set for add/remove
- Don't create unbounded vector clocks — Limit to server nodes
- Don't forget tombstone garbage collection — Clean up old tombstones
- Do consider semantics — Add-wins vs remove-wins
- Do test with concurrent operations — Simulate network partitions
Conflict resolution in practice
Q1/3 You store carts in Riak and want your application to see conflicting versions and merge them itself. Which setting do you need?
Answer & explanation
Answer: A
With siblings enabled, Riak uses vector clocks to detect concurrent writes and returns all of them for the application to resolve.
allow_mult: falsefalls back to last-write-wins, which silently discards one cart.Q2/3 Two users like a post at the same moment. Both read 100 and write 101 to an LWW register, so one like is lost. What is the fix?
Answer & explanation
Answer: C
LWW is wrong for additive operations: both writes are "101", so better timestamps change nothing. A PN- or G-Counter merges both increments to 102. Keeping the larger value still yields 101, because both writers computed the same number from the same stale read.
Q3/3 Your mobile app gives each of a million devices its own vector-clock entry, and clocks become huge and slow to compare. What is the fix?
Answer & explanation
Answer: B
Clock size grows with the number of writers it tracks, so make the servers the writers: a fixed, small number of entries. Truncating entries breaks the happened-before comparison, and switching to LWW reintroduces silent data loss.
Part IV: Interview Preparation#
Chapter 9: Interview Tips#
9.1 Key Phrases#
Introducing conflict resolution
"When we have concurrent updates to the same data, we need a conflict resolution strategy. The choice depends on the data semantics — is latest always correct? Can updates be merged? Is user intervention possible?"
On LWW
"Last-write-wins is the simplest approach — timestamp decides. It works when the latest value is the 'right' value, like a user's current location. But it loses data when concurrent updates both matter, like a shopping cart."
On vector clocks
"Vector clocks track causality — they tell us if one update happened before another, or if they're concurrent. If concurrent, we know we have a conflict that needs resolution. The application can then merge or ask the user."
On CRDTs
"CRDTs are data structures designed to merge automatically. A counter CRDT sums increments from all nodes. A set CRDT takes the union. They're perfect for collaborative apps where you need offline support and automatic sync."
On choosing
"For a shopping cart, I'd use an OR-Set CRDT. Items can be added and removed, and concurrent adds should both appear. For a simple status field, LWW is fine. For a document, I'd use operational transformation or a sequence CRDT."
9.2 Common Questions#
| Question | Good Answer |
|---|---|
| "How do you handle conflicts in a distributed shopping cart?" | "I'd use an OR-Set CRDT. Each item addition creates a unique tag. Removes tombstone existing tags. Concurrent adds from different devices both survive because they have different tags. This gives us add-wins semantics which matches user expectations." |
| "When would you use LWW vs CRDTs?" | "LWW when latest is correct (location, status) or data is regenerable (cache). CRDTs when concurrent updates should merge (counters, sets) or offline support is needed. Vector clocks when I need to detect conflicts for manual resolution." |
| "How do CRDTs guarantee convergence?" | "CRDTs have merge operations that are commutative, associative, and idempotent. This means the order of merges doesn't matter — all replicas reach the same state. G-Counter sums node counts, OR-Set unions with tags, etc." |
| "What's the difference between G-Set and OR-Set?" | "G-Set only allows adds — merge is union. OR-Set allows removes by using unique tags per add. When you remove, you tombstone the tag. Concurrent add of same item creates new tag, so it survives the remove. Add-wins semantics." |
Chapter 10: Practice Problems#
Practice · problem 1
Problem 1: Collaborative Whiteboard#
Setup: Design conflict resolution for a collaborative whiteboard where users can add and move shapes.
Requirements:
- Users can add shapes (rectangle, circle)
- Users can move shapes
- Users can delete shapes
- Works offline
Questions:
- What CRDTs would you use?
- How do you handle concurrent moves of the same shape?
- How do you handle move + delete conflict?
Solution Outline
CRDTs:
- Shape registry: OR-Set of shape IDs
- Shape properties: LWW-Map per shape (x, y, width, height, color)
Concurrent moves:
- Each shape position is LWW-Register
- Latest move wins (acceptable for whiteboard)
- Alternative: Average positions if needed
Move + delete:
- OR-Set handles this with add-wins
- If shape deleted, move is ignored (shape not in set)
- If move happens after delete, shape stays deleted
class WhiteboardCRDT:
def __init__(self, node_id):
self.node_id = node_id
self.shapes = ORSet(node_id) # Set of shape IDs
self.properties: Dict[str, LWWMap] = {} # Per-shape properties
def add_shape(self, shape_id, properties):
self.shapes.add(shape_id)
self.properties[shape_id] = LWWMap(self.node_id)
for k, v in properties.items():
self.properties[shape_id].set(k, v)
def move_shape(self, shape_id, x, y):
if shape_id in self.shapes.value():
self.properties[shape_id].set("x", x)
self.properties[shape_id].set("y", y)
def delete_shape(self, shape_id):
self.shapes.remove(shape_id)Practice · problem 2
Problem 2: Distributed Like Counter#
Setup: Design a like counter for social media posts that works globally.
Requirements:
- Users can like and unlike
- Count should be accurate (no lost likes)
- Low latency reads from any region
Questions:
- What CRDT would you use?
- How do you handle like + unlike from same user?
- How do you handle displaying count efficiently?
Solution Outline
CRDT:
- Use OR-Set for user IDs who liked
- Count = size of set
- Unlike removes user from set
Like + unlike:
- OR-Set handles this correctly
- Add wins over concurrent remove
- Last state wins for sequential like/unlike
Efficient display:
- Cache materialized count locally
- Update count incrementally on merge
- Don't recount set on every read
class LikeCounterCRDT:
def __init__(self, node_id, post_id):
self.likers = ORSet(node_id)
self.post_id = post_id
self._cached_count = 0
def like(self, user_id):
self.likers.add(user_id)
self._update_cache()
def unlike(self, user_id):
self.likers.remove(user_id)
self._update_cache()
def get_count(self) -> int:
return self._cached_count
def _update_cache(self):
self._cached_count = len(self.likers.value())
def merge(self, other):
self.likers.merge(other.likers)
self._update_cache()Chapter 11: Sample Interview Dialogue#
Scenario: Design Offline-First Note Taking App#
"Design a note-taking app that works offline and syncs across devices."
"Before I design, let me understand the conflict scenarios. Users might edit the same note on different devices while offline. When they come online, we need to merge those edits."
"Yes, that's the main challenge."
"For conflict resolution, I have a few options. Let me think through them..."
You sketch on the whiteboard:
1. LWW (Last-Write-Wins)
- Simple but loses edits
- User adds paragraph on phone, laptop edit wins
- BAD for notes
2. Manual merge (show conflict)
- Show both versions, user picks
- Annoying for frequent conflicts
- OK as fallback
3. Automatic merge (CRDT)
- Text CRDT like RGA or Yjs
- Concurrent edits both appear
- BEST for collaborative editing
"I'd use a text CRDT like Yjs or Automerge. Each character has a unique ID based on node and sequence number. Inserts reference the character they come after. Deletes are tombstones."
"How does that handle concurrent edits?"
"Say I type 'Hello' on my phone while you type 'World' on the laptop, both at position 0. Each character gets a unique ID:
Phone: H(phone,1) e(phone,2) l(phone,3) l(phone,4) o(phone,5)
Laptop: W(laptop,1) o(laptop,2) r(laptop,3) l(laptop,4) d(laptop,5)When we merge, the CRDT uses the IDs to determine order. The two concurrent runs both start at position 0, so a deterministic ordering of IDs decides which run goes first; because each character is anchored to the one before it, each run stays contiguous ('HelloWorld' or 'WorldHello'). Both edits appear — no data loss."
"What about the note structure itself, not just the text?"
"For the notes collection, I'd use an OR-Set of note IDs. Each note's metadata (title, tags, folder) uses an LWW-Map — last edit to each field wins. The note content uses the text CRDT.
class NotesCRDT:
def __init__(self, node_id):
self.notes = ORSet(node_id) # Set of note IDs
self.metadata: Dict[str, LWWMap] = {} # Per-note metadata
self.content: Dict[str, TextCRDT] = {} # Per-note contentThis gives us:
- Add/delete notes: OR-Set semantics (add-wins)
- Metadata: LWW per field
- Content: Full text merge"
"What about conflicts in the same line of text?"
"The text CRDT handles this at the character level. If I insert 'A' after position 5 and you insert 'B' after position 5, both characters appear with deterministic ordering. It might look like 'AB' or 'BA' depending on our node IDs, but both edits are preserved.
For a note-taking app, this is usually fine. If users need more control, I could fall back to showing the conflict and letting them choose, but with text CRDTs that's rarely necessary."
Summary#
The conflict problem
- Concurrent updates create conflicts
- Network partitions, offline operation, replication lag
- Must choose resolution strategy
Resolution strategies
- Last-write-wins (LWW)
- Simple: Timestamp wins
- Loses data
- Use for: Location, status, cache
- Vector clocks
- Detect concurrent vs causal
- Application resolves
- Use for: Conflict detection
- CRDTs
- Automatic merge
- No conflicts by design
- Use for: Collaborative, offline-first
CRDT types
- G-Counter: Grow-only counter (sum)
- PN-Counter: Positive-negative counter
- G-Set: Grow-only set (union)
- OR-Set: Observed-remove set (add-wins)
- LWW-Map: Last-writer-wins map
- RGA: Replicated sequence (text)
Choosing strategy
- Latest is correct → LWW
- Need to detect conflicts → Vector clocks
- Can auto-merge → CRDT
- Need user input → Manual resolution
Default approach
- Shopping cart: OR-Set CRDT
- Counters/votes: PN-Counter CRDT
- Simple fields: LWW
- Collaborative text: RGA/Yjs CRDT
- Complex merge: Vector clocks + application logic
Part V: Production Patterns#
Chapter 12: Hybrid Approaches#
12.1 Combining Strategies#
# Hybrid Conflict Resolution System
from dataclasses import dataclass
from typing import Any, Dict, Optional, Callable, List
from enum import Enum
class ResolutionType(Enum):
LWW = "lww"
CRDT = "crdt"
VECTOR_CLOCK = "vector_clock"
CUSTOM = "custom"
@dataclass
class FieldConfig:
"""Configuration for how to resolve conflicts for a field."""
resolution_type: ResolutionType
crdt_type: Optional[str] = None # For CRDT fields
custom_resolver: Optional[Callable] = None # For custom fields
class HybridDocument:
"""
Document with different resolution strategies per field.
Example: User profile
- name: LWW (latest wins)
- email: LWW (latest wins)
- tags: OR-Set CRDT (union of tags)
- view_count: G-Counter CRDT (sum)
- bio: LWW here (a custom merge could surface conflicts for long text)
"""
def __init__(self, node_id: str, field_configs: Dict[str, FieldConfig]):
self.node_id = node_id
self.configs = field_configs
# Storage for different types
self.lww_fields: Dict[str, LWWValue] = {}
self.crdt_fields: Dict[str, Any] = {} # Various CRDT types
self.vector_clock = VectorClock()
def get(self, field: str) -> Any:
"""Get field value."""
config = self.configs.get(field)
if not config:
raise ValueError(f"Unknown field: {field}")
if config.resolution_type == ResolutionType.LWW:
lww = self.lww_fields.get(field)
return lww.value if lww else None
elif config.resolution_type == ResolutionType.CRDT:
crdt = self.crdt_fields.get(field)
if crdt is None:
return self._default_crdt_value(config.crdt_type)
return self._get_crdt_value(crdt, config.crdt_type)
return None
def set(self, field: str, value: Any):
"""Set field value."""
config = self.configs.get(field)
if not config:
raise ValueError(f"Unknown field: {field}")
self.vector_clock = self.vector_clock.increment(self.node_id)
if config.resolution_type == ResolutionType.LWW:
self.lww_fields[field] = LWWValue(
value=value,
timestamp=time.time(),
node_id=self.node_id
)
elif config.resolution_type == ResolutionType.CRDT:
self._update_crdt(field, value, config.crdt_type)
def merge(self, other: 'HybridDocument'):
"""Merge with another document."""
# Merge vector clock
self.vector_clock = self.vector_clock.merge(other.vector_clock)
# Merge each field according to its config
for field, config in self.configs.items():
if config.resolution_type == ResolutionType.LWW:
self._merge_lww(field, other)
elif config.resolution_type == ResolutionType.CRDT:
self._merge_crdt(field, other, config.crdt_type)
elif config.resolution_type == ResolutionType.CUSTOM:
self._merge_custom(field, other, config.custom_resolver)
def _merge_lww(self, field: str, other: 'HybridDocument'):
"""Merge LWW field."""
self_val = self.lww_fields.get(field)
other_val = other.lww_fields.get(field)
if self_val is None:
self.lww_fields[field] = other_val
elif other_val is not None and other_val.timestamp > self_val.timestamp:
self.lww_fields[field] = other_val
def _merge_crdt(self, field: str, other: 'HybridDocument', crdt_type: str):
"""Merge CRDT field."""
self_crdt = self.crdt_fields.get(field)
other_crdt = other.crdt_fields.get(field)
if self_crdt is None:
self.crdt_fields[field] = other_crdt
elif other_crdt is not None:
self_crdt.merge(other_crdt)
def _update_crdt(self, field: str, value: Any, crdt_type: str):
"""Update CRDT field based on type."""
if field not in self.crdt_fields:
self.crdt_fields[field] = self._create_crdt(crdt_type)
crdt = self.crdt_fields[field]
if crdt_type == "g_counter":
crdt.increment(value)
elif crdt_type == "pn_counter":
if value >= 0:
crdt.increment(value)
else:
crdt.decrement(abs(value))
elif crdt_type == "or_set":
crdt.add(value)
elif crdt_type == "lww_map":
for k, v in value.items():
crdt.set(k, v)
def _create_crdt(self, crdt_type: str):
"""Create CRDT of specified type."""
if crdt_type == "g_counter":
return GCounter(self.node_id)
elif crdt_type == "pn_counter":
return PNCounter(self.node_id)
elif crdt_type == "or_set":
return ORSet(self.node_id)
elif crdt_type == "lww_map":
return LWWMap(self.node_id)
raise ValueError(f"Unknown CRDT type: {crdt_type}")
def _get_crdt_value(self, crdt, crdt_type: str):
"""Get value from CRDT."""
if crdt_type in ("g_counter", "pn_counter"):
return crdt.value()
elif crdt_type == "or_set":
return crdt.value()
elif crdt_type == "lww_map":
return crdt.to_dict()
return None
def _default_crdt_value(self, crdt_type: str):
"""Default value for CRDT type."""
if crdt_type in ("g_counter", "pn_counter"):
return 0
elif crdt_type == "or_set":
return set()
elif crdt_type == "lww_map":
return {}
return None
# Example: User Profile
def create_user_profile(node_id: str) -> HybridDocument:
"""Create user profile with hybrid conflict resolution."""
configs = {
"name": FieldConfig(ResolutionType.LWW),
"email": FieldConfig(ResolutionType.LWW),
"bio": FieldConfig(ResolutionType.LWW),
"tags": FieldConfig(ResolutionType.CRDT, crdt_type="or_set"),
"view_count": FieldConfig(ResolutionType.CRDT, crdt_type="g_counter"),
"settings": FieldConfig(ResolutionType.CRDT, crdt_type="lww_map"),
}
return HybridDocument(node_id, configs)
# Usage
profile_us = create_user_profile("us-east")
profile_eu = create_user_profile("eu-west")
# US updates name
profile_us.set("name", "John Doe")
profile_us.set("tags", "developer")
# EU updates (concurrent)
profile_eu.set("name", "John D.") # Different name
profile_eu.set("tags", "python") # Different tag
# Merge
profile_us.merge(profile_eu)
profile_eu.merge(profile_us)
# Result:
# name: "John D." (LWW - EU was later)
# tags: {"developer", "python"} (OR-Set - both tags kept)12.2 Conflict Detection with Resolution Options#
# Conflict Detection and Resolution Service
from dataclasses import dataclass
from typing import List, Optional, Callable
from enum import Enum
class ConflictType(Enum):
NO_CONFLICT = "no_conflict"
CONCURRENT_UPDATE = "concurrent_update"
DELETE_UPDATE = "delete_update"
SEMANTIC_CONFLICT = "semantic_conflict"
@dataclass
class Conflict:
"""Represents a detected conflict."""
type: ConflictType
field: str
local_value: Any
remote_value: Any
local_clock: VectorClock
remote_clock: VectorClock
suggested_resolution: Optional[Any] = None
class ConflictDetector:
"""
Detects and classifies conflicts for resolution.
"""
def __init__(self, semantic_rules: Dict[str, Callable] = None):
self.semantic_rules = semantic_rules or {}
def detect_conflicts(
self,
local: Dict[str, Any],
local_clock: VectorClock,
remote: Dict[str, Any],
remote_clock: VectorClock
) -> List[Conflict]:
"""Detect all conflicts between local and remote state."""
conflicts = []
# Check clock relationship
comparison = local_clock.compare(remote_clock)
if comparison == ClockComparison.BEFORE:
# Local is older - no conflict, just update
return []
if comparison == ClockComparison.AFTER:
# Local is newer - no conflict, remote should update
return []
if comparison == ClockComparison.EQUAL:
# Same version - no conflict
return []
# CONCURRENT - check each field
all_fields = set(local.keys()) | set(remote.keys())
for field in all_fields:
local_val = local.get(field)
remote_val = remote.get(field)
if local_val == remote_val:
continue
conflict = self._classify_conflict(
field, local_val, remote_val,
local_clock, remote_clock
)
if conflict.type != ConflictType.NO_CONFLICT:
conflicts.append(conflict)
return conflicts
def _classify_conflict(
self,
field: str,
local_val: Any,
remote_val: Any,
local_clock: VectorClock,
remote_clock: VectorClock
) -> Conflict:
"""Classify a field conflict."""
# Check for delete vs update
if local_val is None and remote_val is not None:
return Conflict(
type=ConflictType.DELETE_UPDATE,
field=field,
local_value=local_val,
remote_value=remote_val,
local_clock=local_clock,
remote_clock=remote_clock,
suggested_resolution=remote_val # Keep the value
)
if local_val is not None and remote_val is None:
return Conflict(
type=ConflictType.DELETE_UPDATE,
field=field,
local_value=local_val,
remote_value=remote_val,
local_clock=local_clock,
remote_clock=remote_clock,
suggested_resolution=local_val # Keep the value
)
# Check semantic rules
if field in self.semantic_rules:
rule = self.semantic_rules[field]
suggested = rule(local_val, remote_val)
return Conflict(
type=ConflictType.SEMANTIC_CONFLICT,
field=field,
local_value=local_val,
remote_value=remote_val,
local_clock=local_clock,
remote_clock=remote_clock,
suggested_resolution=suggested
)
# Default: concurrent update
return Conflict(
type=ConflictType.CONCURRENT_UPDATE,
field=field,
local_value=local_val,
remote_value=remote_val,
local_clock=local_clock,
remote_clock=remote_clock
)
class ConflictResolver:
"""
Resolves conflicts using various strategies.
"""
def __init__(self):
self.strategies: Dict[str, Callable] = {}
def register_strategy(self, field: str, strategy: Callable):
"""Register a resolution strategy for a field."""
self.strategies[field] = strategy
def resolve(self, conflicts: List[Conflict]) -> Dict[str, Any]:
"""Resolve all conflicts and return merged state."""
resolved = {}
unresolved = []
for conflict in conflicts:
if conflict.suggested_resolution is not None:
resolved[conflict.field] = conflict.suggested_resolution
elif conflict.field in self.strategies:
strategy = self.strategies[conflict.field]
resolved[conflict.field] = strategy(
conflict.local_value,
conflict.remote_value
)
else:
unresolved.append(conflict)
if unresolved:
raise UnresolvedConflictError(unresolved)
return resolved
def auto_resolve(
self,
conflicts: List[Conflict],
default_strategy: str = "lww"
) -> Dict[str, Any]:
"""Auto-resolve conflicts with default strategy."""
resolved = {}
for conflict in conflicts:
if conflict.suggested_resolution is not None:
resolved[conflict.field] = conflict.suggested_resolution
elif conflict.field in self.strategies:
strategy = self.strategies[conflict.field]
resolved[conflict.field] = strategy(
conflict.local_value,
conflict.remote_value
)
elif default_strategy == "lww":
# Use local (assuming local is "latest")
resolved[conflict.field] = conflict.local_value
elif default_strategy == "keep_remote":
resolved[conflict.field] = conflict.remote_value
elif default_strategy == "merge_arrays":
if isinstance(conflict.local_value, list) and isinstance(conflict.remote_value, list):
resolved[conflict.field] = list(set(conflict.local_value) | set(conflict.remote_value))
else:
resolved[conflict.field] = conflict.local_value
return resolved
# Example semantic rules
SEMANTIC_RULES = {
"quantity": lambda local, remote: max(local, remote), # Take larger quantity
"price": lambda local, remote: min(local, remote), # Take lower price
"updated_at": lambda local, remote: max(local, remote), # Latest timestamp
"tags": lambda local, remote: list(set(local) | set(remote)), # Union
}Chapter 13: Tombstone Management#
The replica that came back after ten days
Tombstones are garbage-collected once every node has seen them, or after a 7-day grace period if some node never confirms. A replica was offline for 10 days and now syncs. What happens to items that were removed from the OR-Set while it was away? What should you do with such a replica?
One strong answer
- Resurrection: the other nodes dropped those tombstones after 7 days. The returning replica still holds the items' tags as live, and merging that with peers that no longer remember the tags were removed brings the items back.
- Why GC anyway: without a grace period, one dead node would keep every tombstone forever.
- What to do: treat a replica offline longer than the grace period as stale. Rebuild it from a healthy peer instead of merging its state.
13.1 Garbage Collecting Tombstones#
# Tombstone Garbage Collection
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Dict, Set, Optional
import asyncio
@dataclass
class TombstoneRecord:
"""Record of a tombstone for garbage collection."""
key: str
tag: str
deleted_at: datetime
deleted_by: str
synced_to: Set[str] # Nodes that have seen this tombstone
class TombstoneGC:
"""
Garbage collector for CRDT tombstones.
Problem: OR-Set tombstones accumulate forever
Solution: GC tombstones after all nodes have seen them
Requirements:
- All nodes must see tombstone before GC
- Need to track which nodes have synced
- After a grace period, GC anyway (assumes lagging nodes are dead or rebuilt)
"""
def __init__(
self,
node_id: str,
all_nodes: Set[str],
grace_period: timedelta = timedelta(days=7)
):
self.node_id = node_id
self.all_nodes = all_nodes
self.grace_period = grace_period
self.tombstones: Dict[str, TombstoneRecord] = {}
def record_tombstone(self, key: str, tag: str):
"""Record a new tombstone."""
record = TombstoneRecord(
key=key,
tag=tag,
deleted_at=datetime.now(timezone.utc),
deleted_by=self.node_id,
synced_to={self.node_id}
)
self.tombstones[f"{key}:{tag}"] = record
def mark_synced(self, key: str, tag: str, node_id: str):
"""Mark that a node has seen this tombstone."""
record_key = f"{key}:{tag}"
if record_key in self.tombstones:
self.tombstones[record_key].synced_to.add(node_id)
def get_collectable(self) -> list[TombstoneRecord]:
"""Get tombstones that can be garbage collected."""
collectable = []
now = datetime.now(timezone.utc)
for record in self.tombstones.values():
# Check if all nodes have seen it
if record.synced_to >= self.all_nodes:
collectable.append(record)
continue
# Check if grace period has passed (for dead nodes).
# Trade-off: a replica offline longer than grace_period can
# resurrect the element (cf. Cassandra's gc_grace_seconds)
if now - record.deleted_at > self.grace_period:
collectable.append(record)
return collectable
async def run_gc(self, crdt: ORSet):
"""Run garbage collection on an OR-Set."""
collectable = self.get_collectable()
for record in collectable:
# Remove from CRDT internal state
if record.key in crdt.elements:
if record.tag in crdt.elements[record.key]:
del crdt.elements[record.key][record.tag]
# Clean up empty entries
if not crdt.elements[record.key]:
del crdt.elements[record.key]
# Remove from tombstone tracking
del self.tombstones[f"{record.key}:{record.tag}"]
async def gc_loop(self, crdt: ORSet, interval: int = 3600):
"""Periodic GC loop."""
while True:
await asyncio.sleep(interval)
await self.run_gc(crdt)| Value | |
|---|---|
| Active carts (assumed) | 10 million |
| Item removals per cart per day (assumed) | 2 → 20 million tombstones a day |
Size of one tombstone: node:uuid tag, flag and overhead (assumed) |
~64 bytes → ~1.3 GB a day |
| Held for the 7-day grace period | 7 × 1.3 GB ≈ ~9 GB |
| Never collected, after a year | 365 × 1.3 GB ≈ ~470 GB |
Conclusion: garbage collection bounds tombstones at about a week of removals instead of growing forever. The price is the 7-day limit: any replica offline longer than that must be rebuilt, not merged.
Hybrid resolution and tombstones
Q1/3 In the hybrid user profile, US sets the name to "John Doe" and adds tag "developer" while EU concurrently sets the name to "John D." and adds tag "python". EU's writes are later. What does the merged profile hold?
Answer & explanation
Answer: B
Each field uses its own strategy: the name is LWW, so the later write wins; tags are an OR-Set, so both concurrent adds survive. Treating the whole document as one LWW value would wrongly drop the US tag.
Q2/3 The conflict detector compares a local clock
{us:3, eu:2}with a remote clock{us:2, eu:2}, and thebiofield differs. What does it report?Answer & explanation
Answer: A
Per-field checks only run when the clocks are concurrent. Here local happened after remote, so differing values are just an older copy. Different values alone do not mean a conflict; causality decides that.
Q3/3 Why can't an OR-Set tombstone be garbage-collected as soon as the remove happens?
Answer & explanation
Answer: C
The tombstone is what tells other replicas that the tag is dead. Drop it before everyone has seen it and the item can be resurrected on the next sync. That is why the collector waits until all nodes have synced the tombstone, or a grace period passes.
Further Reading#
Papers#
- Book
"A Comprehensive Study of Convergent and Commutative Replicated Data Types" by Shapiro et al. (2011)
- Book
"Conflict-free Replicated Data Types" by Shapiro et al.
- Read
"Dynamo: Amazon's Highly Available Key-value Store" — Vector clocks section
Libraries#
- Read
Yjs: https://yjs.dev/ — Text CRDT for collaborative editing
- Read
Automerge: https://automerge.org/ — JSON CRDT library
- Read
Riak: https://riak.com/ — Database with CRDT support
Books#
- Book
"Designing Data-Intensive Applications" by Martin Kleppmann — Chapter 5
Videos#
- Talk
Martin Kleppmann CRDTs: Search "Martin Kleppmann CRDT" on YouTube
this lesson in shorts · 2
play them all →One idea from this lesson in about a minute each, made for your phone.
Go deeper · reading trail
all reading →Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.
Amazon's Dynamo
Giuseppe DeCandia, Deniz Hastorun et al. · 2007- What
- Werner Vogels' page publishing the full SOSP 2007 Dynamo paper.
- Key idea
- An always-writable store keeps concurrent versions, tracks them with vector clocks and hands siblings to the application to merge, like a shopping cart union.
- Why here
- The vector clock chapter of this lesson in its most famous production use, including why LWW was not enough.
Roshi: a CRDT system for timestamped events
Peter Bourgon · 2014- What
- SoundCloud's open-source store for the activity stream, built on Redis.
- Key idea
- A last-writer-wins element set CRDT keeps add and remove sets with timestamps, so replicas merge in any order and converge.
- Why here
- A small, readable production CRDT, and an example of LWW done per element rather than per object.
How Figma's multiplayer technology works
Evan Wallace · 2019- What
- Figma's explanation of how simultaneous edits to one design file are merged.
- Key idea
- Figma rejected operational transforms; inspired by CRDTs, a central server orders changes and conflicts resolve last-writer-wins per object property.
- Why here
- A worked 'choose the right approach' decision from this lesson: when a central authority exists, a simpler CRDT-like scheme beats full OT.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.

