~learn.
?

Day 0346 minPreface+60 XP

Rate Limiting at Scale.

5 stages37 code9 Qs

watch it first · newThe whole lesson in a 18-minute animated lecture5 quiz checkpoints · ask the AI about any moment · captions, chapters and speed

Preface · 1 min

Preface#

Days 1 and 2 taught you how to store and replicate data. Today, you learn how to protect those systems from being overwhelmed.

Rate limiting sounds simple: count requests, reject when over limit. But at scale, rate limiting becomes a distributed systems problem with no perfect solution. You'll face questions like:

  • How do you count requests across 50 servers?
  • What happens when your rate limit store fails?
  • Is it better to accidentally allow too many requests, or accidentally block legitimate ones?

By the end of this session, you'll understand the algorithms, the distributed coordination challenges, and the failure modes that make rate limiting one of the most deceptively complex components in modern systems.

Let's begin.

ExplainerThe whole lesson: rate limiting at scale

Rate limits defend against botnets, runaway scripts, surprise bills and unfair sharing. A fixed window lets 10 requests at 00:55 and 10 at 01:05 through (20 in 10 seconds), a sliding log is exact but stores every timestamp, a sliding counter estimates 8 × 75% + 3 = 9 with two counters, and a token bucket (10 a second, capacity 50) allows bursts. Across 50 servers uncoordinated counters let 100 requests past a limit of 10, so counts go to a central Redis (+1 to 2 ms, a single point of failure), local counters with sync, or sticky servers; the design (10,000 req/s, 100 per user and 1,000 per key a minute, under 5 ms) returns 429 with Retry-After, and when one of three Redis nodes dies (about 33% of counters, 10 to 30 s of failover) it falls back to conservative local limits instead of failing fully open or closed.

Part I · Stage 1 of 5 · 17 min

Part I: Foundations#

Chapter 1: Why Rate Limiting Matters#

1.1 The Threats You're Defending Against#

Rate limiting protects your system from several classes of problems:

Threat 1: Denial of Service (DoS)#

A malicious actor floods your API with requests, consuming all available resources. Legitimate users can't get through.

Without rate limiting One bad actor with a botnet can take down your entire service.

With rate limiting Each IP/user is limited to N requests per second. The attacker is throttled; legitimate users continue normally.

Threat 2: Resource Exhaustion#

A single customer accidentally runs a script that makes 1 million API calls in a loop. They didn't mean to, but they're consuming resources meant for all customers.

Without rate limiting One runaway script degrades service for everyone.

With rate limiting The script hits its limit, gets throttled, and the customer fixes their code.

Threat 3: Cost Control#

Your service calls expensive downstream APIs (payment processors, AI models, third-party data). Each call costs money.

Without rate limiting A bug or abuse could generate a massive bill.

With rate limiting You cap the maximum spend per customer per time period.

Threat 4: Fair Resource Allocation#

You have 100 customers sharing infrastructure. One customer is 10x larger than the others. Without limits, they'd consume most resources.

With rate limiting: Each customer gets a fair share (or a share proportional to their plan).

1.2 The Rate Limiting Contract#

A rate limit is a contract with your users:

"You can make up to N requests per time period T. If you exceed this, we'll reject additional requests until the period resets."

Example contracts:

  • 100 requests per minute per API key
  • 1000 requests per hour per IP address
  • 10 requests per second per user (with bursting up to 50)

The contract must be:

  • Clear: Users know their limits before hitting them
  • Predictable: Same behavior every time
  • Fair: Applied consistently across all users
  • Communicated: Response headers tell users their remaining quota

1.3 Where Rate Limiting Lives#

Rate limiting can be implemented at multiple layers:

DiagramRate limiting at every layer
%% title: Rate limiting at every layer
flowchart TD
  I@{ icon: "sd:globe", label: "Internet" } --> CDN@{ icon: "sd:cdn", label: "CDN / Edge<br/>Layer 1: Geographic rate limiting,<br/>DDoS protection" }
  CDN --> LB@{ icon: "sd:load-balancer", label: "Load Balancer<br/>Layer 2: Connection limits,<br/>basic throttling" }
  LB --> GW@{ icon: "sd:gateway", label: "API Gateway<br/>Layer 3: Per-API-key limits, quotas" }
  GW --> APP["Application<br/>Layer 4: Business logic limits<br/>(e.g., 5 password attempts)"]:::core
  APP --> DB@{ icon: "sd:database", label: "Database<br/>Layer 5: Connection pools,<br/>query limits" }
  class I client
  class CDN,LB,GW edge
  class DB store

Today, we focus on Layers 3 and 4: API-level rate limiting that requires distributed coordination.

Chapter 2: Rate Limiting Algorithms#

There are five major algorithms for rate limiting. Each has different characteristics for burstiness, memory usage, and accuracy.

2.1 Fixed Window Counter#

How It Works#

Divide time into fixed windows (e.g., 1-minute intervals). Count requests in each window. Reset the counter when the window changes.

  • Window 1 (00:00-00:59): [||||||||||] 10 requests → Allow
  • Window 2 (01:00-01:59): [||||||||||||||||||||] 20 requests → Allow 10, reject 10
  • Window 3 (02:00-02:59): [|||] 3 requests → Allow

Implementation:

pythonFixedWindowLimiter · 21 lines
class FixedWindowLimiter:
    def __init__(self, limit: int, window_seconds: int):
        self.limit = limit
        self.window_seconds = window_seconds
        self.counters: dict[str, tuple[int, int]] = {}  # key -> (count, window_start)
    
    def is_allowed(self, key: str) -> bool:
        now = int(time.time())
        window_start = now - (now % self.window_seconds)
        
        count, stored_window = self.counters.get(key, (0, window_start))
        
        # New window - reset counter
        if stored_window != window_start:
            count = 0
            stored_window = window_start
        
        if count < self.limit:
            self.counters[key] = (count + 1, stored_window)
            return True
        return False
Strengths#
  • Simple: Easy to understand and implement
  • Memory efficient: One counter per key per window
  • Fast: O(1) lookup and update
Weaknesses#
  • Boundary problem: A user can make 2x the limit at the boundary between windows
Boundary problem — limit: 10 requests per minute
  • Window 1 (00:00-00:59) —
    User makes 10 requests at 00:55
  • Window 2 (01:00-01:59) —
    User makes 10 requests at 01:05

Result: 20 requests in 10 seconds (00:55 to 01:05). But each window only saw 10 → both allowed!

This "burst at the boundary" can double the effective rate.

2.2 Sliding Window Log#

How It Works#

Store the timestamp of each request. To check if a new request is allowed, count requests in the past window (sliding).

WorksheetSliding window log
  • Limit: 10 requests per 60 seconds
  • Current time: 12:05:30
  • Request log for user: [12:04:25, 12:04:40, 12:05:00, 12:05:10, 12:05:15] — 12:04:25 older than 60s ago, ignored
  • Requests in window (12:04:30 to 12:05:30): 4
  • New request allowed: Yes (4 < 10)

Implementation:

pythonSlidingWindowLogLimiter · 19 lines
class SlidingWindowLogLimiter:
    def __init__(self, limit: int, window_seconds: int):
        self.limit = limit
        self.window_seconds = window_seconds
        self.logs: dict[str, list[float]] = {}
    
    def is_allowed(self, key: str) -> bool:
        now = time.time()
        window_start = now - self.window_seconds
        
        # Get existing timestamps, filter out old ones
        timestamps = self.logs.get(key, [])
        timestamps = [t for t in timestamps if t > window_start]
        
        if len(timestamps) < self.limit:
            timestamps.append(now)
            self.logs[key] = timestamps
            return True
        return False
Strengths#
  • Accurate: No boundary problem—the window truly slides
  • Precise: Exact count of requests in any rolling window
Weaknesses#
  • Memory intensive: Must store timestamp of every request
  • Cleanup overhead: Need to periodically remove old timestamps
  • Slow at scale: Counting requests requires scanning the log

For high-volume APIs (thousands of requests per second per key), this becomes expensive.

Your turn · checkpoint 1not answered

Estimate it: two counters instead of a log

Limit: 10 requests per 60 seconds. It's 12:01:15. The previous fixed window (12:00) saw 8 requests; the current one (12:01) has 3 so far. Without storing timestamps, estimate how many requests fall in the last 60 seconds. Is the next request allowed?

Think it through, or jot it down. Then compare.

One strong answer

  • Overlap: the sliding window 12:00:15–12:01:15 still covers 45 of the previous window's 60 seconds, so count 75% of it.
  • Estimate: 8 × 0.75 + 3 = 6 + 3 = 9, under 10, so the request is allowed.
  • The trade: two counters per key instead of a timestamp per request, at the price of an estimate rather than an exact count.

2.3 Sliding Window Counter#

How It Works#

A hybrid approach: use fixed windows but weight them based on how much of the current sliding window they cover.

WorksheetSliding window counter
  • Limit: 10 requests per 60 seconds
  • Current time: 12:01:15 (15 seconds into the new minute)
  • Previous window (12:00:00-12:00:59): 8 requests
  • Current window (12:01:00-12:01:59): 3 requests (so far)
  • Weighted count = (previous × overlap%) + current = (8 × 75%) + 3 = 6 + 3 = 9
  • 9 < 10 → Request allowed

The "overlap%" is how much of the previous window is still within our 60-second sliding window:

  • At 12:01:15, we're 15 seconds into the new window
  • So the previous window contributes 75% (45 seconds of overlap / 60 seconds)

Implementation:

pythonSlidingWindowCounterLimiter · 34 lines
class SlidingWindowCounterLimiter:
    def __init__(self, limit: int, window_seconds: int):
        self.limit = limit
        self.window_seconds = window_seconds
        self.counters: dict[str, dict] = {}
    
    def is_allowed(self, key: str) -> bool:
        now = time.time()
        current_window = int(now // self.window_seconds)
        window_position = now % self.window_seconds
        
        data = self.counters.get(key, {
            'prev_count': 0,
            'prev_window': current_window - 1,
            'curr_count': 0,
            'curr_window': current_window
        })
        
        # Handle window transitions
        if data['curr_window'] < current_window:
            data['prev_count'] = data['curr_count'] if data['curr_window'] == current_window - 1 else 0
            data['prev_window'] = data['curr_window']
            data['curr_count'] = 0
            data['curr_window'] = current_window
        
        # Calculate weighted count
        prev_weight = (self.window_seconds - window_position) / self.window_seconds
        weighted_count = (data['prev_count'] * prev_weight) + data['curr_count']
        
        if weighted_count < self.limit:
            data['curr_count'] += 1
            self.counters[key] = data
            return True
        return False
Strengths#
  • Accurate enough: Smooths the boundary problem without storing every timestamp
  • Memory efficient: Only two counters per key
  • Fast: O(1) operations
Weaknesses#
  • Approximate: Not perfectly accurate (weighted estimate, not exact count)
  • Slightly more complex: Than fixed window

This is the sweet spot for most production systems.

Your turn · checkpoint 2not answered

Your call: let bursts through, stop steady spam

Your users go quiet for a while, then fire a handful of requests at once, and that's normal. A spammer sends at a steady high rate. You want bursts allowed up to a cap, and the long-run rate held to 10 a second. What state would you keep per key, and how would it decide?

Think it through, or jot it down. Then compare.

One strong answer

  • A token bucket: a token count and a last-update time. Tokens refill at 10 a second up to a capacity (say 50); each request spends one, and an empty bucket rejects.
  • Bursts up to the capacity pass after quiet periods; the long-run rate can't exceed the refill rate, so steady spam runs dry.
  • Bonus: an expensive operation can cost several tokens.

2.4 Token Bucket#

How It Works#

Imagine a bucket that holds tokens. The bucket fills at a constant rate (e.g., 10 tokens per second). Each request consumes one token. If the bucket is empty, the request is rejected.

The bucket has a maximum capacity, allowing for controlled bursting.

Token bucket — 10 tokens/second rate, 50 token capacity
  • Time 0 —
    Bucket has 50 tokens (full). 30 requests arrive → 30 allowed, 20 tokens remain
  • Time 1 —
    Bucket refills 10 tokens → 30 tokens. 5 requests arrive → 5 allowed, 25 tokens remain
  • Time 2 —
    Bucket refills 10 tokens → 35 tokens. 0 requests → 35 tokens remain
  • Time 3 —
    Bucket refills 10 tokens → 45 tokens. 60 requests → 45 allowed, 15 rejected, 0 tokens remain

Implementation:

pythonTokenBucketLimiter · 22 lines
class TokenBucketLimiter:
    def __init__(self, rate: float, capacity: int):
        self.rate = rate  # tokens per second
        self.capacity = capacity
        self.buckets: dict[str, tuple[float, float]] = {}  # key -> (tokens, last_update)
    
    def is_allowed(self, key: str, tokens_requested: int = 1) -> bool:
        now = time.time()
        
        current_tokens, last_update = self.buckets.get(key, (self.capacity, now))
        
        # Add tokens based on time elapsed
        elapsed = now - last_update
        current_tokens = min(self.capacity, current_tokens + (elapsed * self.rate))
        
        if current_tokens >= tokens_requested:
            current_tokens -= tokens_requested
            self.buckets[key] = (current_tokens, now)
            return True
        else:
            self.buckets[key] = (current_tokens, now)
            return False
Strengths#
  • Allows bursting: Users can burst up to bucket capacity
  • Smooth rate over time: Long-term rate converges to the configured rate
  • Flexible: Can charge different costs for different operations
Weaknesses#
  • Bursting can be problematic: If all users burst simultaneously, you might exceed capacity
  • Slightly more state: Need to track tokens and timestamp per key
ExplainerBreak it: token bucket vs a burst

Sliders for bucket capacity, refill rate and the burst at t3, simulated second by second as in the lesson (refill up to capacity, then serve). With 10 tokens/s and capacity 50, requests of 30, 5, 0 and 60 followed by a steady 10/s: all 30 pass at t0, and at t3 45 of the 60 pass and 15 are rejected. A smaller capacity rejects more of any burst; a refill rate below the steady 10/s rejects requests every second.

2.5 Leaky Bucket#

How It Works#

Requests enter a bucket (queue). The bucket "leaks" at a constant rate—requests are processed at a fixed rate regardless of arrival pattern.

Leaky bucket — process 10 requests/second, queue capacity 50
  • Burst —
    Requests arrive in burst: [30 requests at once]. Queue: [30 requests waiting]. Processing: 10/second
  • After 1 second —
    10 processed, 20 in queue
  • After 2 seconds —
    20 processed, 10 in queue
  • After 3 seconds —
    30 processed, queue empty
Strengths#
  • Smooth output rate: Downstream systems see a constant load
  • Prevents bursting: Output is always at the configured rate
Weaknesses#
  • Adds latency: Requests wait in queue
  • Queue management: Need to handle full queue (reject new requests)
  • Not suitable for real-time: Users wait even when system has capacity

Leaky bucket is common for smoothing traffic to downstream services, not for user-facing rate limits.

ExplainerToken bucket vs leaky bucket

Token bucket (10 tokens/s, capacity 50): a full bucket lets a burst of 30 through at once, refills 10 per second and rejects what it can't cover. Leaky bucket (10/s, queue 50): the same burst of 30 waits in the queue and leaves at a steady 10 per second.

2.6 Algorithm Comparison#

Algorithm Accuracy Memory Burstiness Best For
Fixed Window Low Very Low High (2x at boundary) Simple use cases
Sliding Window Log Exact High None When accuracy is critical
Sliding Window Counter Good Low Low Most API rate limits
Token Bucket Configurable Low Controlled Allowing bursts, varied costs
Leaky Bucket N/A Medium None Smoothing to downstream

Recommendation: Start with sliding window counter for API rate limiting. Use token bucket if you need controlled bursting.

Chapter 3: Distributed Rate Limiting#

Everything so far assumed a single server. In reality, you have 50 servers, and requests from the same user can hit any of them.

3.1 The Coordination Problem#

Without coordination

User makes 100 requests. Limit: 10 requests/minute.

Server Sees Decision
Server 1 2 requests Allows all
Server 2 3 requests Allows all
Server 3 2 requests Allows all
...
Server 50 2 requests Allows all

Total allowed: 100 (should have been 10!)

Each server only sees a fraction of the traffic. Without coordination, users can exceed limits by spreading requests across servers.

Your turn · checkpoint 3not answered

Fifty servers, one limit

A user's requests land on any of 50 app servers. Name three ways to make them count against one limit, and what each costs.

Think it through, or jot it down. Then compare.

One strong answer

  • A central counter in Redis: accurate, but every check adds a round trip (+1–2 ms), and Redis becomes a dependency that can fail.
  • Local counters, synced periodically: fast, and they keep working briefly without Redis, but users can exceed the global limit between syncs.
  • Sticky routing by user: each server owns its users' counters, with no coordination; but load is uneven, and a server failure disrupts its users.

3.2 Solution 1: Centralized Counter (Redis)#

Store counters in a central store that all servers access.

DiagramCentralized counter store
%% title: Centralized counter store
flowchart TD
  subgraph AS["Application Servers"]
    direction TB
    S1@{ icon: "sd:server", label: "S1" }
    S2@{ icon: "sd:server", label: "S2" }
    S3@{ icon: "sd:server", label: "S3" }
    SD["..."]:::note
    S50@{ icon: "sd:server", label: "S50" }
  end
  S1 --> R@{ icon: "tech:redis", label: "Redis Cluster<br/>(Counters)" }
  S2 --> R
  S3 --> R
  S50 --> R
  class S1,S2,S3,S50 core
  class R cache

Implementation with Redis:

pythonRedisRateLimiter · 53 lines
import redis
import time

class RedisRateLimiter:
    def __init__(self, redis_client: redis.Redis, limit: int, window_seconds: int):
        self.redis = redis_client
        self.limit = limit
        self.window_seconds = window_seconds
    
    def is_allowed(self, key: str) -> tuple[bool, dict]:
        """
        Returns (allowed, info) where info contains remaining quota and reset time.
        Uses the sliding window log algorithm (one sorted-set entry per request).
        """
        now = time.time()
        window_key = f"ratelimit:{key}"
        
        # Use Redis pipeline for atomic operations
        pipe = self.redis.pipeline()
        
        # Remove old entries (outside window)
        window_start = now - self.window_seconds
        pipe.zremrangebyscore(window_key, 0, window_start)
        
        # Count current entries
        pipe.zcard(window_key)
        
        # Add current request (optimistically)
        pipe.zadd(window_key, {f"{now}:{id(now)}": now})
        
        # Set expiry on the key
        pipe.expire(window_key, self.window_seconds)
        
        results = pipe.execute()
        current_count = results[1]
        
        if current_count < self.limit:
            remaining = self.limit - current_count - 1
            return True, {
                'allowed': True,
                'remaining': max(0, remaining),
                'reset': int(now + self.window_seconds),
                'limit': self.limit
            }
        else:
            # Over limit - remove the optimistically added entry
            self.redis.zrem(window_key, f"{now}:{id(now)}")
            return False, {
                'allowed': False,
                'remaining': 0,
                'reset': int(now + self.window_seconds),
                'limit': self.limit
            }

Optimized Token Bucket in Redis (Lua script):

lua34 lines
-- Token bucket rate limiter as a Lua script (atomic execution)
-- KEYS[1] = rate limit key
-- ARGV[1] = rate (tokens per second)
-- ARGV[2] = capacity (max tokens)
-- ARGV[3] = current timestamp
-- ARGV[4] = tokens requested

local key = KEYS[1]
local rate = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

-- Get current state
local data = redis.call('HMGET', key, 'tokens', 'last_update')
local tokens = tonumber(data[1]) or capacity
local last_update = tonumber(data[2]) or now

-- Calculate token refill
local elapsed = now - last_update
local new_tokens = math.min(capacity, tokens + (elapsed * rate))

-- Check if request can be fulfilled
local allowed = 0
if new_tokens >= requested then
    new_tokens = new_tokens - requested
    allowed = 1
end

-- Update state
redis.call('HMSET', key, 'tokens', new_tokens, 'last_update', now)
redis.call('EXPIRE', key, math.ceil(capacity / rate) * 2)

return {allowed, new_tokens, capacity}
pythonRedisTokenBucket · 16 lines
class RedisTokenBucket:
    LUA_SCRIPT = """..."""  # The Lua script above
    
    def __init__(self, redis_client: redis.Redis, rate: float, capacity: int):
        self.redis = redis_client
        self.rate = rate
        self.capacity = capacity
        self.script = self.redis.register_script(self.LUA_SCRIPT)
    
    def is_allowed(self, key: str, tokens: int = 1) -> tuple[bool, float]:
        now = time.time()
        result = self.script(
            keys=[f"tokenbucket:{key}"],
            args=[self.rate, self.capacity, now, tokens]
        )
        return bool(result[0]), result[1]  # (allowed, remaining_tokens)
Strengths#
  • Accurate: All servers share the same counter
  • Simple model: Easy to reason about
Weaknesses#
  • Single point of failure: If Redis goes down, what happens?
  • Latency: Every request requires a Redis round-trip
  • Scalability ceiling: Redis cluster has limits

3.3 Solution 2: Local Counters with Synchronization#

Each server maintains local counters, periodically syncing with the central store.

DiagramLocal counters with periodic sync
%% title: Local counters with periodic sync
flowchart TD
  S1["Server 1<br/>Local count: 5<br/>Last sync: 10 sec ago"]:::core
  S2["Server 2<br/>Local count: 3<br/>Last sync: 5 sec ago"]:::core
  S1 -.->|"Periodic sync"| R[("Redis<br/>Global count")]:::cache
  S2 -.->|"Periodic sync"| R

Implementation:

pythonLocalSyncRateLimiter · 49 lines
import threading
import time

class LocalSyncRateLimiter:
    def __init__(
        self,
        redis_client: redis.Redis,
        local_limit: int,
        global_limit: int,
        window_seconds: int,
        sync_interval: float = 1.0
    ):
        self.redis = redis_client
        self.local_limit = local_limit  # Per-server allowance
        self.global_limit = global_limit
        self.window_seconds = window_seconds
        self.sync_interval = sync_interval
        
        self.local_counts: dict[str, int] = {}
        self.lock = threading.Lock()
        
        # Start sync thread
        self.sync_thread = threading.Thread(target=self._sync_loop, daemon=True)
        self.sync_thread.start()
    
    def is_allowed(self, key: str) -> bool:
        with self.lock:
            count = self.local_counts.get(key, 0)
            if count < self.local_limit:
                self.local_counts[key] = count + 1
                return True
            return False
    
    def _sync_loop(self):
        while True:
            time.sleep(self.sync_interval)
            self._sync_to_redis()
    
    def _sync_to_redis(self):
        with self.lock:
            counts_to_sync = self.local_counts.copy()
            self.local_counts.clear()
        
        pipe = self.redis.pipeline()
        for key, count in counts_to_sync.items():
            redis_key = f"ratelimit:{key}"
            pipe.incrby(redis_key, count)
            pipe.expire(redis_key, self.window_seconds)
        pipe.execute()
Strengths#
  • Lower latency: Local checks are fast
  • Resilient: Can continue (briefly) if Redis is unavailable
Weaknesses#
  • Approximate: Can exceed global limits between syncs
  • More complex: Need to tune local vs global limits
  • Coordination: Server count affects local limits

3.4 Solution 3: Sticky Sessions#

Route requests from the same user to the same server. Each server handles a subset of users.

DiagramSticky routing by user
%% title: Sticky routing by user
flowchart TD
  LB["Load Balancer<br/>(sticky sessions by user_id)"]:::edge
  LB --> S1["Server1<br/>User A<br/>User D"]:::core
  LB --> S2["Server2<br/>User B<br/>User E"]:::core
  LB --> S3["Server3<br/>User C<br/>User F"]:::core

Each server tracks limits only for its assigned users—no coordination needed!

Strengths#
  • No coordination: Each server is authoritative for its users
  • Fast: No remote calls for rate limiting
  • Simple: Local data structures only
Weaknesses#
  • Uneven load: Some users are heavier than others
  • Failover complexity: What happens when a server dies?
  • Not always possible: Some architectures can't do sticky sessions

3.5 Comparing Distributed Approaches#

Approach Accuracy Latency Failure Mode Complexity
Centralized Redis High +1-2ms Total failure if Redis down Low
Local + Sync Approximate Very low Continues with local limits Medium
Sticky Sessions High (per server) Very low Failover disruption Low
Hybrid High Low Graceful degradation High

Chapter 4: Failure Modes and Graceful Degradation#

The most important question in distributed rate limiting:

When your rate limit store fails, do you over-allow (let requests through) or under-allow (reject requests)?

Your turn · checkpoint 4not answered

Redis is down: open or closed?

Your rate-limit store is unreachable. For each limit, do you let requests through or reject them: API limits for paying customers, calls to an expensive AI model billed per request, DDoS protection, login attempts?

Think it through, or jot it down. Then compare.

One strong answer

  • Paying customers' API limits: over-allow. Don't lose customers to your own outage.
  • Expensive AI calls: under-allow. A runaway bill is worse than a few blocked calls.
  • DDoS protection: under-allow. Better to block everyone than let the attack through.
  • Login: under-allow. An unprotected window invites brute force.
  • Middle ground: fall back to conservative local limits, so nobody is fully blocked and nothing is fully unprotected.

4.1 The Over-Allow vs Under-Allow Trade-off#

Over-allow (fail open):

  • If Redis is down, allow all requests
  • Pro: Legitimate users aren't blocked
  • Con: No protection during outage; abuse possible

Under-allow (fail closed):

  • If Redis is down, reject all requests
  • Pro: System protected from overload
  • Con: Legitimate users blocked during outage

Neither is universally correct. The choice depends on what you're protecting.

4.2 Decision Framework#

Scenario Recommendation Reasoning
DDoS protection Under-allow Better to block everyone than let attack through
API monetization Over-allow Don't lose paying customers due to your bug
Resource protection Under-allow Protect backend systems from overload
User-facing rate limits Over-allow User experience > strict enforcement
Cost control (expensive APIs) Under-allow Financial risk of over-allowing is high

4.3 Implementing Graceful Degradation#

Hybrid approach: Local limits as fallback when centralized store fails.

pythonResilientRateLimiter · 61 lines
class ResilientRateLimiter:
    def __init__(
        self,
        redis_client: redis.Redis,
        global_limit: int,
        local_fallback_limit: int,
        window_seconds: int,
        failure_mode: str = 'over_allow'  # or 'under_allow'
    ):
        self.redis = redis_client
        self.global_limit = global_limit
        self.local_fallback_limit = local_fallback_limit
        self.window_seconds = window_seconds
        self.failure_mode = failure_mode
        
        self.local_limiter = SlidingWindowCounterLimiter(
            local_fallback_limit, window_seconds
        )
        self.redis_available = True
        self.last_redis_check = 0
        self.redis_check_interval = 5  # seconds
    
    def is_allowed(self, key: str) -> tuple[bool, str]:
        """Returns (allowed, mode) where mode is 'redis', 'local', or 'failed'."""
        
        # Try Redis first
        if self._redis_is_available():
            try:
                return self._check_redis(key), 'redis'
            except redis.RedisError:
                self.redis_available = False
        
        # Redis unavailable - use fallback strategy
        if self.failure_mode == 'over_allow':
            # Use local limiter as soft limit
            allowed = self.local_limiter.is_allowed(key)
            return allowed, 'local'
        else:
            # Fail closed - but maybe allow some traffic
            if self.local_fallback_limit > 0:
                # Very conservative local limit
                allowed = self.local_limiter.is_allowed(key)
                return allowed, 'local_strict'
            return False, 'failed'
    
    def _redis_is_available(self) -> bool:
        now = time.time()
        if not self.redis_available:
            # Periodically retry Redis
            if now - self.last_redis_check > self.redis_check_interval:
                self.last_redis_check = now
                try:
                    self.redis.ping()
                    self.redis_available = True
                except redis.RedisError:
                    pass
        return self.redis_available
    
    def _check_redis(self, key: str) -> bool:
        # Implement your Redis rate limit check here
        ...

4.4 Monitoring Rate Limiter Health#

You need to know when your rate limiter is degraded:

pythonInstrumentedRateLimiter · 33 lines
from prometheus_client import Counter, Histogram, Gauge

# Metrics
rate_limit_decisions = Counter(
    'rate_limit_decisions_total',
    'Rate limit decisions',
    ['result', 'mode']  # result=allowed/denied, mode=redis/local/failed
)

rate_limit_latency = Histogram(
    'rate_limit_check_seconds',
    'Time to check rate limit',
    buckets=[.001, .005, .01, .025, .05, .1, .25, .5, 1]
)

redis_available = Gauge(
    'rate_limiter_redis_available',
    'Whether Redis is available for rate limiting'
)

class InstrumentedRateLimiter:
    def __init__(self, limiter: ResilientRateLimiter):
        self.limiter = limiter
    
    def is_allowed(self, key: str) -> bool:
        with rate_limit_latency.time():
            allowed, mode = self.limiter.is_allowed(key)
        
        result = 'allowed' if allowed else 'denied'
        rate_limit_decisions.labels(result=result, mode=mode).inc()
        redis_available.set(1 if self.limiter.redis_available else 0)
        
        return allowed

Key metrics to alert on:

  • Redis availability dropping
  • Rate limit latency increasing
  • Unusual spike in denials
  • Mode switching from 'redis' to 'local'
Quick check4 questions · +5 XP each

Rate limiting algorithms and distribution

  1. Q1/4 With a fixed window of 10 requests per minute, a client sends 10 at 00:55 and 10 at 01:05, and all 20 are allowed. Which algorithm fixes this most cheaply?

    Answer & explanation

    Answer: A

    The sliding window counter smooths the boundary with just two counters per key, the lesson's default for most APIs. A sliding window log is exact but stores a timestamp per request; a bigger fixed window still has a boundary.

  2. Q2/4 Sliding window counter: the limit is 100 per minute. You are 15 seconds into the current window, which has 45 requests; the previous window had 80. Is the next request allowed?

    Answer & explanation

    Answer: B

    The previous window still overlaps 45 of the last 60 seconds: 80 x 0.75 + 45 = 60 + 45 = 105, which is over 100. Counting only the current window is the fixed-window behavior that allows boundary bursts.

  3. Q3/4 Fifty app servers each keep an in-memory counter with a limit of 10 per minute per user. A user spreads 100 requests evenly across them. What happens?

    Answer & explanation

    Answer: C

    Each server sees a fraction of the traffic, so no single counter reaches 10. Shared state (e.g. Redis) or sticky routing is needed. Load balancers just route; they do not know the per-user limit.

  4. Q4/4 Redis, your rate-limit store, goes down. Which limit should fail closed (reject requests)?

    Answer & explanation

    Answer: B

    Under-allow when over-allowing costs real money or risks overload. A runaway bill is worse than a few blocked calls. For user-facing, non-critical limits the lesson prefers over-allowing so legitimate users are not blocked by your outage.

Part II · Stage 2 of 5 · 7 min

Part II: The Design Challenge#

Chapter 5: Designing a Distributed Rate Limiter#

You're designing a rate limiting service for your platform. Requirements:

  • Scale: 10,000 requests per second across 50 application servers
  • Limits: Per-user limits (100 req/min), per-API-key limits (1000 req/min), global limits
  • Latency: Rate limit check must add < 5ms to request latency (p99)
  • Accuracy: Within 10% of configured limits
  • Resilience: Continue operating if Redis has a brief outage
WorksheetThe limiter's load (rules per request, entry size and active users are assumptions)
Value
Traffic (requirement) 10,000 requests/s over 50 servers → 200/s per server
Rules checked per request (assumed: per-user, per-API-key, global) 3 → ~30,000 checks/s in Redis
Redis round trip (+1–2 ms, from 3.5) 3 checks one after another: up to ~6 ms, over the 5 ms p99 budget
Sliding-window-log entries held (one per request in the last 60 s) 10,000 × 60 = 600,000
Bytes per stored entry or counter (assumed) ~100 → the log takes ~60 MB
Sliding window counters (assumed 50,000 active users × 2 counters) ~10 MB
One of 3 Redis nodes down for ~30 s (the failover in 5.5) 10,000 × 30 ÷ 3 = ~100,000 decisions on the local fallback

Conclusion: memory isn't the constraint at this scale; latency and failure are. The checks for one request must share a single Redis round trip (a pipeline or one Lua script, as the lesson's Redis code does), and the fallback must be designed, because a single node failure hands it ~100,000 decisions.

5.1 Architecture Overview#

DiagramRate limit service architecture
%% title: Rate limit service architecture
flowchart TD
  REQ@{ icon: "sd:users", label: "Request from Client" } --> CS
  subgraph RLS["Rate Limit Service"]
    direction TB
    CS@{ icon: "sd:database", label: "Configuration Store<br/>(Limit rules, per-tier configs)" }
    subgraph LOGIC["Rate Limit Logic"]
      direction TB
      LC@{ icon: "sd:cache", label: "Local Cache" } <--> RC@{ icon: "tech:redis", label: "Redis Cluster" }
    end
    DEC{"Decision<br/>Allow / Deny"}:::hot
    CS --> LOGIC
    LOGIC --> DEC
  end
  class REQ client
  class CS store
  class LC,RC cache

5.2 Data Model#

Rate limit rules (stored in configuration):

pythonRateLimitRule · 54 lines
@dataclass
class RateLimitRule:
    id: str
    name: str
    key_pattern: str  # e.g., "user:{user_id}", "api_key:{api_key}"
    limit: int
    window_seconds: int
    algorithm: str  # 'sliding_window' or 'token_bucket'
    burst_capacity: Optional[int]  # For token bucket
    scope: str  # 'per_user', 'per_api_key', 'global'
    tier: Optional[str]  # 'free', 'pro', 'enterprise'
    
# Example rules
rules = [
    RateLimitRule(
        id="user_basic",
        name="Basic user limit",
        key_pattern="user:{user_id}",
        limit=100,
        window_seconds=60,
        algorithm="sliding_window",
        scope="per_user",
        tier="free"
    ),
    RateLimitRule(
        id="user_pro",
        name="Pro user limit",
        key_pattern="user:{user_id}",
        limit=1000,
        window_seconds=60,
        algorithm="sliding_window",
        scope="per_user",
        tier="pro"
    ),
    RateLimitRule(
        id="api_key",
        name="API key limit",
        key_pattern="apikey:{api_key}",
        limit=10000,
        window_seconds=3600,
        algorithm="token_bucket",
        burst_capacity=500,
        scope="per_api_key"
    ),
    RateLimitRule(
        id="global_writes",
        name="Global write limit",
        key_pattern="global:writes",
        limit=5000,
        window_seconds=1,
        algorithm="sliding_window",
        scope="global"
    )
]

5.3 Multi-Level Rate Limiting#

A single request might be subject to multiple limits:

pythonMultiLevelRateLimiter · 71 lines
class MultiLevelRateLimiter:
    def __init__(self, redis_client: redis.Redis, rules: list[RateLimitRule]):
        self.redis = redis_client
        self.rules = rules
        self.limiters = self._create_limiters()
    
    def _create_limiters(self) -> dict[str, RateLimiter]:
        limiters = {}
        for rule in self.rules:
            if rule.algorithm == 'sliding_window':
                limiters[rule.id] = RedisSlidingWindowLimiter(
                    self.redis, rule.limit, rule.window_seconds
                )
            elif rule.algorithm == 'token_bucket':
                limiters[rule.id] = RedisTokenBucket(
                    self.redis, 
                    rate=rule.limit / rule.window_seconds,
                    capacity=rule.burst_capacity or rule.limit
                )
        return limiters
    
    def check_request(self, context: RequestContext) -> RateLimitResult:
        """
        Check all applicable rules for this request.
        Returns the most restrictive result.
        """
        applicable_rules = self._get_applicable_rules(context)
        
        results = []
        for rule in applicable_rules:
            key = self._build_key(rule, context)
            limiter = self.limiters[rule.id]
            
            allowed, info = limiter.is_allowed(key)
            results.append(RateLimitResult(
                rule_id=rule.id,
                allowed=allowed,
                limit=rule.limit,
                remaining=info.get('remaining', 0),
                reset=info.get('reset', 0)
            ))
        
        # If any rule denies, the request is denied
        denied_results = [r for r in results if not r.allowed]
        if denied_results:
            # Return the result with the longest wait time
            return max(denied_results, key=lambda r: r.reset)
        
        # All rules allow - return the one with lowest remaining quota
        return min(results, key=lambda r: r.remaining)
    
    def _get_applicable_rules(self, context: RequestContext) -> list[RateLimitRule]:
        applicable = []
        for rule in self.rules:
            # Check tier match
            if rule.tier and rule.tier != context.user_tier:
                continue
            # Check scope
            if rule.scope == 'per_user' and not context.user_id:
                continue
            if rule.scope == 'per_api_key' and not context.api_key:
                continue
            applicable.append(rule)
        return applicable
    
    def _build_key(self, rule: RateLimitRule, context: RequestContext) -> str:
        key = rule.key_pattern
        key = key.replace('{user_id}', context.user_id or 'anonymous')
        key = key.replace('{api_key}', context.api_key or 'none')
        key = key.replace('{ip}', context.ip_address)
        return key

5.4 Response Headers#

Communicate rate limit status to clients:

pythonadd_rate_limit_headers · 7 lines
def add_rate_limit_headers(response: Response, result: RateLimitResult):
    response.headers['X-RateLimit-Limit'] = str(result.limit)
    response.headers['X-RateLimit-Remaining'] = str(result.remaining)
    response.headers['X-RateLimit-Reset'] = str(result.reset)
    
    if not result.allowed:
        response.headers['Retry-After'] = str(result.reset - int(time.time()))

Common (de facto) headers:

  • X-RateLimit-Limit: The rate limit ceiling
  • X-RateLimit-Remaining: Requests remaining in current window
  • X-RateLimit-Reset: Unix timestamp when the limit resets
  • Retry-After: Seconds until client should retry (on 429)
Your turn · checkpoint 5not answered

What breaks if one Redis node dies?

The limiter's counters live in a 3-node Redis Cluster, and one node dies. Which users are affected, for how long, and what should the limiter do for them? What's still wrong after failover?

Think it through, or jot it down. Then compare.

One strong answer

  • Affected: only keys on the dead node, about a third of users. Everyone else carries on normally.
  • For how long: until the cluster promotes a replica, roughly 10–30 seconds.
  • What to do: fall back to a conservative local limit for those keys. Nobody is blocked and the system keeps some protection, at the cost of some over-allowing.
  • After failover: if the replica was behind, recent increments are gone, so some counters read a little low: a mini-reset you accept.

5.5 The Challenge Question: Redis Node Dies#

Scenario: You have a 3-node Redis cluster. One node dies. What happens to rate limiting?

Analysis:

  1. Keys on the dead node are temporarily inaccessible

    • In Redis Cluster, keys are distributed across nodes
    • ~33% of keys are on the dead node
    • Those rate limit counters are unavailable
  2. Cluster failover takes time

    • Redis Cluster automatic failover: 10-30 seconds
    • During this time, reads/writes to affected keys fail
  3. Options during the outage:

    Option A: Fail open for affected keys

    pythonis_allowed · 6 lines
    def is_allowed(self, key: str) -> bool:
        try:
            return self._check_redis(key)
        except redis.ClusterDownError:
            # Redis node down - allow the request
            return True

    Option B: Fail closed for affected keys

    pythonis_allowed · 6 lines
    def is_allowed(self, key: str) -> bool:
        try:
            return self._check_redis(key)
        except redis.ClusterDownError:
            # Redis node down - deny the request
            return False

    Option C: Fall back to local rate limiting

    pythonis_allowed · 6 lines
    def is_allowed(self, key: str) -> bool:
        try:
            return self._check_redis(key)
        except redis.ClusterDownError:
            # Use conservative local limit
            return self.local_limiter.is_allowed(key)
  4. After failover completes:

    • If replica was in sync, no data loss
    • If replica was behind, some recent increments lost
    • Rate limits might be slightly "reset" for affected keys

My recommendation: Option C (local fallback) with conservative limits. During the ~30 second outage:

  • 66% of keys (other nodes) continue normally
  • 33% of keys (affected node) use local fallback
  • No users are completely blocked
  • Some users might slightly exceed limits
  • System remains available

5.6 Complete Implementation#

pythonRateLimitMiddleware · 54 lines
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import JSONResponse
import redis
from typing import Optional

app = FastAPI()

class RateLimitMiddleware:
    def __init__(self, app: FastAPI, limiter: MultiLevelRateLimiter):
        self.app = app
        self.limiter = limiter
    
    async def __call__(self, request: Request, call_next):
        # Extract context from request
        context = RequestContext(
            user_id=request.headers.get('X-User-ID'),
            api_key=request.headers.get('X-API-Key'),
            ip_address=request.client.host,
            user_tier=request.headers.get('X-User-Tier', 'free'),
            endpoint=request.url.path,
            method=request.method
        )
        
        # Check rate limits
        result = self.limiter.check_request(context)
        
        if not result.allowed:
            response = JSONResponse(
                status_code=429,
                content={
                    'error': 'rate_limit_exceeded',
                    'message': f'Rate limit exceeded. Retry after {result.reset - int(time.time())} seconds.',
                    'limit': result.limit,
                    'reset': result.reset
                }
            )
            add_rate_limit_headers(response, result)
            return response
        
        # Proceed with request
        response = await call_next(request)
        add_rate_limit_headers(response, result)
        return response


# Setup
redis_client = redis.RedisCluster(
    host='redis-cluster.example.com',
    port=6379,
    decode_responses=True
)

limiter = MultiLevelRateLimiter(redis_client, rules)
app.middleware('http')(RateLimitMiddleware(app, limiter))
Quick check3 questions · +5 XP each

Designing the distributed rate limiter

  1. Q1/3 In a 3-node Redis Cluster used for rate limiting, one node dies. During the roughly 30-second failover, what does the lesson recommend?

    Answer & explanation

    Answer: C

    Only keys on the dead node are unavailable. A local fallback keeps those users served with some protection, while everyone else is unaffected. Rejecting everything blocks the 66% of users whose counters are fine.

  2. Q2/3 A request passes its per-user limit but fails the global write limit. What does the multi-level limiter return?

    Answer & explanation

    Answer: A

    Every applicable rule is checked, and the request needs all of them to allow it. A global limit protects the system no matter how many quotas individual users have left.

  3. Q3/3 After failover, the promoted Redis replica was slightly behind. What effect does that have on rate limiting?

    Answer & explanation

    Answer: A

    An async replica misses the last few increments, so affected counters read a bit low. The lesson accepts this rather than blocking everyone. Nothing is wiped; counters are just slightly behind.

Part III · Stage 3 of 5 · 3 min

Part III: Advanced Topics#

Chapter 6: Rate Limiting Patterns#

6.1 Tiered Rate Limits#

Different limits for different user tiers:

pythonget_limits_for_user · 20 lines
TIER_LIMITS = {
    'free': {
        'requests_per_minute': 60,
        'requests_per_day': 1000,
        'burst': 10
    },
    'pro': {
        'requests_per_minute': 600,
        'requests_per_day': 50000,
        'burst': 100
    },
    'enterprise': {
        'requests_per_minute': 6000,
        'requests_per_day': None,  # Unlimited
        'burst': 1000
    }
}

def get_limits_for_user(user: User) -> dict:
    return TIER_LIMITS.get(user.tier, TIER_LIMITS['free'])

6.2 Endpoint-Specific Limits#

Different endpoints have different costs:

pythonCostAwareRateLimiter · 14 lines
ENDPOINT_COSTS = {
    'GET /api/users': 1,
    'POST /api/users': 5,
    'GET /api/reports': 10,  # Expensive query
    'POST /api/ai/generate': 100,  # Very expensive
}

class CostAwareRateLimiter:
    def __init__(self, limiter: TokenBucketLimiter):
        self.limiter = limiter
    
    def check_request(self, user_id: str, endpoint: str) -> bool:
        cost = ENDPOINT_COSTS.get(endpoint, 1)
        return self.limiter.is_allowed(user_id, tokens_requested=cost)

6.3 Adaptive Rate Limiting#

Adjust limits based on system load:

pythonAdaptiveRateLimiter · 20 lines
class AdaptiveRateLimiter:
    def __init__(self, base_limiter: RateLimiter, load_monitor: LoadMonitor):
        self.limiter = base_limiter
        self.load_monitor = load_monitor
    
    def get_effective_limit(self, base_limit: int) -> int:
        load = self.load_monitor.get_current_load()  # 0.0 to 1.0
        
        if load < 0.5:
            # Low load - allow 20% more
            return int(base_limit * 1.2)
        elif load < 0.8:
            # Normal load - use base limit
            return base_limit
        elif load < 0.95:
            # High load - reduce by 20%
            return int(base_limit * 0.8)
        else:
            # Critical load - reduce by 50%
            return int(base_limit * 0.5)

6.4 Rate Limit Exemptions#

Some requests should bypass rate limits:

pythonshould_exempt · 23 lines
EXEMPT_PATTERNS = [
    '/health',
    '/metrics',
    '/api/webhooks/stripe',  # Stripe webhook callbacks
]

EXEMPT_API_KEYS = {
    'internal-service-key-123',
    'monitoring-key-456',
}

def should_exempt(request: Request) -> bool:
    # Check path exemptions
    for pattern in EXEMPT_PATTERNS:
        if request.url.path.startswith(pattern):
            return True
    
    # Check API key exemptions
    api_key = request.headers.get('X-API-Key')
    if api_key in EXEMPT_API_KEYS:
        return True
    
    return False

Chapter 7: Rate Limiting at the Edge#

7.1 CDN-Level Rate Limiting#

For DDoS protection, rate limit at the edge:

DiagramRate limiting at the edge
%% title: Rate limiting at the edge
flowchart TD
  I@{ icon: "sd:globe", label: "Internet" } --> CDN
  subgraph CDN["CDN / Edge Network"]
    direction TB
    P1@{ icon: "sd:cdn", label: "Edge POP 1<br/>(Tokyo)" }
    P2@{ icon: "sd:cdn", label: "Edge POP 2<br/>(London)" }
    P3@{ icon: "sd:cdn", label: "Edge POP 3<br/>(NYC)" }
    PD["..."]:::note
    RL{"Rate Limit Check<br/>(per IP, per region, global)"}:::core
    P1 --> RL
    P2 --> RL
    P3 --> RL
  end
  RL --> O@{ icon: "sd:servers", label: "Origin Servers" }
  class I client
  class P1,P2,P3 edge
  class O service

Cloudflare rate limiting rules (example):

Rule Action Scope
Block if > 100 requests in 10 seconds from same IP Block for 1 hour All endpoints
Challenge if > 1000 requests in 1 minute from same /24 subnet JavaScript challenge Login endpoints

7.2 API Gateway Rate Limiting#

AWS API Gateway, Kong, or similar:

yaml12 lines
# Kong rate limiting configuration
plugins:
  - name: rate-limiting
    config:
      minute: 100
      hour: 1000
      policy: redis
      redis_host: redis.example.com
      redis_port: 6379
      redis_database: 0
      fault_tolerant: true  # Continue if Redis fails
      hide_client_headers: false

Benefits:

  • No application code changes
  • Centralized configuration
  • Can rate limit before hitting your servers

Limitations:

  • Less flexibility than application-level
  • Can't easily do user-tier-based limits without custom integration
Quick check2 questions · +5 XP each

Rate limiting patterns and the edge

  1. Q1/2 A cost-aware token bucket holds 500 tokens and POST /api/ai/generate costs 100. The user's bucket is full and they fire 6 calls back to back. How many succeed?

    Answer & explanation

    Answer: B

    Five calls spend all 500 tokens; the sixth needs 100 more than the bucket has. Charging per-endpoint costs is a strength of token bucket. Bursting is allowed, but only up to the bucket's capacity.

  2. Q2/2 A botnet floods your login endpoint from thousands of IPs worldwide. Where should the first line of rate limiting sit?

    Answer & explanation

    Answer: C

    Edge rate limiting (per IP, subnet, region) absorbs a DDoS close to the source, so your servers never see most of it. By the time the app or database is limiting, the flood is already using your capacity.

Part IV · Stage 4 of 5 · 3 min

Part IV: Discussion and Trade-offs#

Chapter 8: The Hard Questions#

8.1 "One Redis node dies—what's the behavior?"#

Complete answer:

"When a Redis node in a cluster dies, keys hashed to that node are temporarily unavailable. Here's my approach:

First, understand the impact: with a 3-node cluster, ~33% of keys are affected. Rate limit checks for those keys will fail.

For the ~30-second failover window, I'd use a fallback strategy:

python4 lines
try:
    return redis_limiter.is_allowed(key)
except redis.ClusterDownError:
    return local_fallback_limiter.is_allowed(key)

The local fallback uses a conservative limit—maybe 10% of the global limit per server. With 50 servers, that's 500% of the global limit in total, so we might over-allow 5x during the outage. That's acceptable for 30 seconds.

After failover, the new primary takes over. If it was slightly behind the failed primary, we might have lost some recent increments—effectively a mini-reset of rate limits for those keys. I'd accept this rather than blocking all requests.

The key is: monitor this scenario. Alert when Redis cluster is degraded. Log when fallback mode activates. Review after every incident."

8.2 "Would you over-allow or under-allow during failures?"#

Complete answer:

"It depends on what the rate limit is protecting.

I'd over-allow for:

  • User-facing API limits (don't block paying customers)
  • General abuse prevention (attackers can wait 30 seconds anyway)
  • Non-critical endpoints

I'd under-allow for:

  • Cost control on expensive downstream APIs (financial risk)
  • Resource protection when the system is already stressed
  • Security-sensitive endpoints (login, password reset)

For our platform, I'd configure per-rule behavior:

pythonRateLimitRule · 3 lines
class RateLimitRule:
    ...
    failure_mode: str  # 'open' or 'closed'

Most rules would be 'open'. But the rule limiting calls to our AI model (which costs $0.01 per call) would be 'closed'—I'd rather block users than accidentally run up a huge bill.

The hybrid approach—local conservative limits during outage—gives a middle ground. Users aren't completely blocked, but we're not completely unprotected either."

8.3 "How do you prevent abuse of the rate limit system itself?"#

Potential attacks:

  1. Key enumeration: Attacker tries to fill up rate limit storage with millions of unique keys
  2. Clock manipulation: If using client timestamps, attacker sends fake times
  3. Distributed attack: Attacker uses many IPs to stay under per-IP limits

Defenses:

pythonSecureRateLimiter · 23 lines
# Defense against key enumeration
class SecureRateLimiter:
    def __init__(self, max_unique_keys: int = 1_000_000):
        self.max_unique_keys = max_unique_keys
    
    def is_allowed(self, key: str) -> bool:
        # Use server time, not client time
        now = time.time()
        
        # Limit total keys tracked (with LRU eviction)
        if self.key_count() > self.max_unique_keys:
            self.evict_oldest_keys()
        
        # For anonymous users, use IP + user agent hash
        # This makes it harder to generate unique keys
        ...

# Defense against distributed attacks
class IPSubnetLimiter:
    def is_allowed(self, ip: str) -> bool:
        # Limit per /24 subnet, not just per IP
        subnet = self.get_subnet(ip, prefix_length=24)
        return self.limiter.is_allowed(f"subnet:{subnet}")

Chapter 9: Session Summary#

What You Should Know Now#

After this session, you should be able to:

  1. Explain the five main algorithms (fixed window, sliding window log, sliding window counter, token bucket, leaky bucket) with trade-offs
  2. Design distributed rate limiting using Redis or local+sync approaches
  3. Handle failure modes with explicit over-allow or under-allow decisions
  4. Implement multi-level rate limits (per-user, per-API-key, per-endpoint)
  5. Monitor and debug rate limiting systems

Key Trade-offs to Remember#

Decision Trade-off
Fixed vs Sliding window Simplicity vs Accuracy at boundaries
Token bucket vs Sliding window Allows bursting vs Strict limits
Centralized vs Local counters Accuracy vs Latency/Resilience
Over-allow vs Under-allow User experience vs Protection
Per-IP vs Per-user limits Coverage vs Fairness to shared IPs

Questions to Ask in Every Design#

  1. What are we protecting? (Resources? Cost? Fairness?)
  2. What's the acceptable accuracy? (Exact? Within 10%?)
  3. What happens during Redis failure?
  4. How do we communicate limits to users?
  5. How do we handle different user tiers?
Quick check3 questions · +5 XP each

Rate limiter failures and abuse

  1. Q1/3 During a Redis outage, each of 50 servers falls back to a local limit of 10% of the global limit. Worst case, how much traffic can a single user get through?

    Answer & explanation

    Answer: B

    50 servers x 10% = 500% of the limit if a user spreads requests across every server. The lesson accepts this for a ~30-second window. Each server only knows its own share, which is why a local fallback over-allows.

  2. Q2/3 An attacker rotates through many IPs in the same range to stay under your per-IP limit. Which defense from the lesson targets this?

    Answer & explanation

    Answer: B

    Aggregating by subnet catches requests from many addresses in one range. Never trust client timestamps; that is itself an attack vector. Raising limits helps the attacker.

  3. Q3/3 The rate-limit store is down. Which endpoint does the lesson say to under-allow (fail closed) on?

    Answer & explanation

    Answer: A

    Security-sensitive endpoints are where an unprotected window invites brute force, so you block rather than allow. For general and paying-customer APIs, the lesson leans toward over-allowing so your outage does not lock them out.

Part V · Stage 5 of 5 · 8 min

Part V: Interview Questions and Answers#

Chapter 10: Real-World Interview Scenarios#

10.1 Conceptual Questions#

Interview question 1not answered
Question 1: "Explain the token bucket algorithm."#

Interviewer's Intent: Testing fundamental algorithm knowledge.

Answer it out loud first. Then compare.

Strong Answer:

"Token bucket is a rate limiting algorithm that allows controlled bursting. Imagine a bucket that holds tokens—it fills at a constant rate (say, 10 tokens per second) up to a maximum capacity (say, 100 tokens).

Each request consumes tokens from the bucket. If there are enough tokens, the request is allowed and tokens are deducted. If not, the request is rejected.

The key properties:

  • Long-term rate is bounded by the fill rate (10/sec)
  • Short-term bursts are allowed up to bucket capacity (100)
  • After a burst, you have to wait for the bucket to refill

Implementation-wise, you don't actually simulate filling—you calculate tokens based on time elapsed:

python1 line
tokens = min(capacity, tokens + (elapsed_time * rate))

I'd use token bucket when I want to allow bursting, like a user who's been idle for a while making several rapid requests. For strict rate enforcement without bursting, I'd use sliding window counter instead."

Interview question 2not answered
Question 2: "What's the boundary problem with fixed window rate limiting?"#

Interviewer's Intent: Testing understanding of algorithm edge cases.

Answer it out loud first. Then compare.

Strong Answer:

"Fixed window rate limiting divides time into fixed intervals—say, minute boundaries. You count requests in each minute and reset at the boundary.

The problem: a user can make their full quota at the end of one window and again at the start of the next window, effectively doubling their rate.

Example with 100 requests/minute limit:

  • 99 requests at 12:00:59 (Window 1)
  • 99 requests at 12:01:01 (Window 2)

That's 198 requests in 2 seconds, way over the intended rate.

Sliding window counter fixes this by weighting the previous window's count based on how much of it overlaps with the current sliding window. If we're 30 seconds into the new window, we count 50% of the previous window's requests plus all of the current window's requests.

In practice, I'd always use sliding window counter for user-facing rate limits. Fixed window is only acceptable for very rough limiting where 2x burst at boundaries is tolerable."

Interview question 3not answered
Question 3: "How would you rate limit across multiple servers?"#

Interviewer's Intent: Testing distributed systems thinking.

Answer it out loud first. Then compare.

Strong Answer:

"The challenge is that without coordination, each server only sees a fraction of requests. A user could exceed limits by spreading requests across servers.

Three approaches:

Centralized store (Redis) All servers check/update a shared counter. Accurate but adds latency and creates a dependency.

Local + periodic sync Each server has local counters, periodically syncing to Redis. Lower latency, but accuracy suffers between syncs.

Sticky sessions Route each user to a consistent server. No coordination needed, but uneven load distribution and failover complexity.

I'd typically use Redis with the sliding window counter algorithm. The latency hit (~1-2ms) is acceptable for most APIs. For ultra-low-latency requirements, I'd use local counters with sync, accepting that we might over-allow by up to (number of servers × local limit) during the sync interval.

The key is building in graceful degradation: if Redis fails, fall back to local limits rather than failing completely open or closed."

10.2 Design Questions#

Interview question 4not answered
Question 4: "Design a rate limiter for a multi-tenant SaaS API."#

Interviewer's Intent: Testing end-to-end system design.

Answer it out loud first. Then compare.

Strong Answer:

"Let me understand the requirements. Multi-tenant means different customers with potentially different limits. I'll assume we have tiers: free (100 req/min), pro (1000 req/min), and enterprise (custom).

Architecture:

DiagramMulti-tenant rate limiter request path
%% title: Multi-tenant rate limiter request path
flowchart TD
  REQ["Request"]:::client --> IT["Identify Tenant"]:::core
  IT --> LL[("Look Up Limits")]:::store
  LL --> CC[("Check Counter")]:::cache
  CC --> AD{"Allow/Deny"}:::hot

Tenant identification From API key in header. Each API key maps to a tenant and tier.

Limit storage I'd store limit configurations in PostgreSQL with caching:

pythonTenantConfig · 4 lines
class TenantConfig:
    tenant_id: str
    tier: str
    custom_limits: dict  # For enterprise overrides

Counter storage: Redis with sliding window counter. Keys like ratelimit:{tenant_id}:{endpoint_group}.

Multi-level limits:

  • Per-tenant overall (1000/min)
  • Per-tenant per-endpoint (e.g., 100/min for writes)
  • Global system protection (100K/min total across all tenants)

Response 429 Too Many Requests with headers showing limit, remaining, and reset time.

Failure handling If Redis is down, fall back to local counters. Free tier gets strict local limits (fail closed to protect resources). Paid tiers get lenient local limits (fail open to avoid blocking paying customers).

Monitoring Track rate limit hits per tenant, alert on high rejection rates, dashboard showing top consumers."

Interview question 5not answered
Question 5: "Design rate limiting for a real-time chat application."#

Interviewer's Intent: Testing adaptation to specific use case.

Answer it out loud first. Then compare.

Strong Answer:

"Chat has different requirements than typical APIs. Users expect instant message delivery, and messages come in bursts (typing a conversation).

Key considerations:

  • Users chat in bursts—strict per-second limits would feel terrible
  • Different limits for different actions (messages, media uploads, connection attempts)
  • Abuse vectors: spam, flooding a room, repeated connection attempts

My approach:

Token bucket for messages: Fill rate 1 token/second, capacity 20 tokens. Users can burst 20 messages quickly (normal conversation), but sustained spam is blocked.

python5 lines
message_limits = {
    'direct_message': TokenBucket(rate=1, capacity=20),
    'group_message': TokenBucket(rate=0.5, capacity=10),  # Stricter for groups
    'media_upload': TokenBucket(rate=0.1, capacity=3),  # Much stricter
}

Fixed window for connections: Limit reconnection attempts to prevent connection-spam attacks:

python1 line
connection_limit = FixedWindow(limit=10, window_minutes=1)

Per-room limits: In addition to per-user limits, limit the total message rate per room to prevent coordinated spam:

python1 line
room_limit = SlidingWindow(limit=100, window_seconds=10)

Implementation Local limits on each chat server (which maintains WebSocket connections), with async sync to Redis for cross-server visibility. The local-first approach is crucial for low latency—we can't add a Redis round trip to every message.

User experience Instead of hard rejection, I'd queue messages that exceed soft limits and deliver them slowly. Only hard reject for severe abuse."

10.3 Scenario-Based Questions#

Interview question 6not answered
Question 6: "Your rate limiter is rejecting legitimate users during traffic spikes. What do you do?"#

Interviewer's Intent: Testing operational problem-solving.

Answer it out loud first. Then compare.

Strong Answer:

"First, triage: Is this affecting all users or specific ones? Are limits correctly configured? Is it actually the rate limiter or something else?

Investigation steps:

  1. Check rate limiter metrics: What's the rejection rate? Which rules are triggering?

  2. Look at specific users: Are they actually exceeding limits, or is the rate limiter malfunctioning?

  3. Check Redis health: High latency or failures could cause incorrect counts.

Immediate mitigations (in order of risk):

  1. Increase limits temporarily: If legitimate traffic grew, raise limits to accommodate:

    bash1 line
    redis-cli SET ratelimit:config:user_basic:limit 200  # Double the limit
  2. Add more headroom per tier: Maybe our limits were too conservative.

  3. Switch to local-only limiting temporarily: If Redis is the bottleneck, local limits might be less restrictive.

  4. Exempt specific users/endpoints: If one critical customer is blocked, add them to exemption list while we fix things.

Post-incident:

  1. Review whether limits match actual usage patterns
  2. Consider adaptive rate limiting that adjusts to traffic patterns
  3. Add alerting for rejection rate spikes
  4. Maybe implement graduated responses: soft limit (warn) before hard limit (block)"
Interview question 7not answered
Question 7: "How would you test your rate limiter?"#

Interviewer's Intent: Testing quality engineering mindset.

Answer it out loud first. Then compare.

Strong Answer:

"Testing rate limiters requires validating both correctness and behavior under failure.

Unit tests for algorithms:

pythontest_sliding_window_allows_under_limit · 18 lines
def test_sliding_window_allows_under_limit():
    limiter = SlidingWindowLimiter(limit=10, window_seconds=60)
    for _ in range(10):
        assert limiter.is_allowed('user1') == True
    assert limiter.is_allowed('user1') == False

def test_sliding_window_resets():
    limiter = SlidingWindowLimiter(limit=10, window_seconds=1)
    for _ in range(10):
        limiter.is_allowed('user1')
    time.sleep(1)
    assert limiter.is_allowed('user1') == True

def test_fixed_window_boundary_burst():
    # Verify we understand the boundary problem
    limiter = FixedWindowLimiter(limit=10, window_seconds=60)
    # Simulate requests at window boundary
    ...

Integration tests with Redis:

pythontest_distributed_counting · 11 lines
def test_distributed_counting():
    # Simulate multiple servers
    limiter1 = RedisRateLimiter(redis_client, limit=10, window_seconds=60)
    limiter2 = RedisRateLimiter(redis_client, limit=10, window_seconds=60)
    
    for _ in range(5):
        limiter1.is_allowed('user1')
        limiter2.is_allowed('user1')
    
    # Both should see the same count
    assert limiter1.is_allowed('user1')[0] == False  # is_allowed returns (allowed, info)

Load tests:

python6 lines
# Hit the rate limiter with 10K requests/second
# Verify:
# - Latency stays under 5ms p99
# - Correct rejection rate (within 10% of expected)
# - No memory leaks
# - Redis connection pool is healthy

Chaos tests:

pythontest_redis_failure_recovery · 7 lines
def test_redis_failure_recovery():
    # Make requests (should use Redis)
    # Kill Redis
    # Make requests (should use fallback)
    # Verify fallback behavior matches config (open/closed)
    # Restart Redis
    # Make requests (should use Redis again)

Production testing:

  • Shadow mode: Log what would be rejected without actually rejecting
  • Canary: Route 1% of traffic to new limiter version
  • Gradual rollout with kill switch"

10.4 Deep-Dive Questions#

Interview question 8not answered
Question 8: "Compare rate limiting implementations in Redis vs in-memory."#

Interviewer's Intent: Testing infrastructure trade-off knowledge.

Answer it out loud first. Then compare.

Strong Answer:

"Redis-based:

Pros:

  • Shared state across all servers (accurate global limits)
  • Persistent (survives server restarts)
  • Atomic operations (Lua scripts for complex logic)
  • Scales independently of application servers

Cons:

  • Network latency (1-2ms per check)
  • Single point of failure (even clustered)
  • Operational overhead
  • Cost at scale

In-memory:

Pros:

  • Near-zero latency (~microseconds)
  • No external dependencies
  • Simple deployment
  • No network failure modes

Cons:

  • Per-server limits only (inaccurate global limits)
  • State lost on restart
  • Scales with application servers (might not want that)
  • Each server uses memory

When to use each:

Redis: User-facing API limits where accuracy matters, multi-tenant systems, when you need persistence.

In-memory: Ultra-low-latency requirements, per-connection limits (like WebSocket messages), local caching layer before Redis.

Hybrid: In-memory first pass (reject obvious violations), Redis for border cases. Or in-memory with periodic Redis sync.

For most web APIs, Redis with proper fallback is the right choice. The 1-2ms overhead is negligible compared to typical API latencies."

Interview question 9not answered
Question 9: "How do services like Cloudflare and AWS do rate limiting at scale?"#

Interviewer's Intent: Testing knowledge of real-world systems.

Answer it out loud first. Then compare.

Strong Answer:

"Cloudflare:

They rate limit at the edge—in their 300+ data centers worldwide. Each edge PoP counts and decides locally; its counters are not shared with other PoPs.

For per-IP limits, this works well—most requests from one IP hit the same PoP. For per-user limits (requiring API key lookup), they need to route to origin or use distributed state.

Cloudflare's own write-up describes an isolated counting system inside each PoP (a memcached cluster behind Twemproxy) and a sliding-window approximation, deliberately avoiding centralized aggregation. The key insight is that for DDoS protection, exact accuracy isn't needed—blocking 90% of an attack is almost as good as blocking 100%.

AWS API Gateway:

Uses token bucket algorithm. Regional limits are enforced within each AWS region. For global limits across regions, you'd need to build your own coordination.

They integrate with AWS WAF for additional IP-based limiting. The rate limiting state is stored in their internal distributed systems, not exposed to users.

Stripe:

Their engineering blog describes a Redis-backed token bucket request limiter, plus a concurrent-requests limiter and load shedders that drop low-priority traffic first. They're notable for clear documentation and error messages, and some endpoints carry their own, stricter limits.

Common patterns:

  1. Edge limiting for DDoS (approximate, fast)
  2. Application limiting for quotas (accurate, slower)
  3. Hierarchical limits (IP < User < Account < Global)
  4. Graceful degradation during partial failures

The theme is: different types of limits at different layers, accepting that distributed rate limiting is inherently approximate."

Chapter 11: Interview Preparation Checklist#

Before your interview, make sure you can:

Algorithms#

Algorithms0/3 ready
  • Explain fixed window, sliding window log, sliding window counter, token bucket
  • Describe the boundary problem and how sliding window solves it
  • Know when to use token bucket vs sliding window

Distributed Systems#

Distributed Systems0/3 ready
  • Design rate limiting with Redis
  • Explain the coordination problem across multiple servers
  • Implement fallback strategies for Redis failure

Failure Modes#

Failure Modes0/3 ready
  • Articulate over-allow vs under-allow trade-off
  • Design graceful degradation
  • Know what to monitor and alert on

Real-World Knowledge#

Real-World Knowledge0/3 ready
  • Know how CDNs/API Gateways do rate limiting
  • Understand multi-tenant rate limiting
  • Cite real examples (Stripe, Cloudflare)

Exercises · 1 min

Exercises#

Exercise 1: Algorithm Implementation#

Implement a sliding window counter rate limiter that:

  • Supports configurable window size and limit
  • Returns remaining quota and reset time
  • Handles concurrent access safely

Exercise 2: Distributed Design#

Design a rate limiting system for a video streaming service:

  • 1M concurrent users
  • Limits on: video starts (5/min), API calls (100/min), bandwidth (10GB/day)
  • Global and per-user limits
  • Handle CDN edge servers and origin servers

Exercise 3: Failure Analysis#

Your rate limiter uses a 3-node Redis cluster. Analyze what happens in each scenario:

  1. One node becomes slow (100ms latency)
  2. One node dies and fails over (30 second outage)
  3. Network partition isolates your app servers from Redis
  4. Redis runs out of memory

For each, describe the user experience and your recommended handling.

Further Reading#

  • Read

    "Design a Rate Limiter" chapter in System Design Interview (Alex Xu, Vol. 1)

  • Docs

    Stripe's API Rate Limiting: Excellent documentation and implementation

  • Blog

    Cloudflare Blog: Posts on edge rate limiting and DDoS protection

  • Docs

    Redis Rate Limiting Patterns: Official Redis documentation

  • Docs

    Kong Rate Limiting Plugin: Open-source API gateway approach

Appendix · 6 min

Appendix: Code Reference#

A.1 Complete Redis Rate Limiter#

pythonRateLimitAlgorithm · 247 lines
import redis
import time
from dataclasses import dataclass
from typing import Optional, Tuple
from enum import Enum

class RateLimitAlgorithm(Enum):
    SLIDING_WINDOW = "sliding_window"
    TOKEN_BUCKET = "token_bucket"

@dataclass
class RateLimitResult:
    allowed: bool
    limit: int
    remaining: int
    reset: int
    retry_after: Optional[int] = None

class ProductionRateLimiter:
    """
    Production-ready rate limiter with:
    - Multiple algorithm support
    - Graceful degradation
    - Comprehensive metrics
    - Multi-key support
    """
    
    SLIDING_WINDOW_SCRIPT = """
    local key = KEYS[1]
    local now = tonumber(ARGV[1])
    local window = tonumber(ARGV[2])
    local limit = tonumber(ARGV[3])
    
    local window_start = now - window
    redis.call('ZREMRANGEBYSCORE', key, 0, window_start)
    local count = redis.call('ZCARD', key)
    
    if count < limit then
        redis.call('ZADD', key, now, now .. ':' .. math.random())
        redis.call('EXPIRE', key, window)
        return {1, limit - count - 1, math.ceil(now + window)}
    else
        local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
        local reset = oldest[2] and (oldest[2] + window) or (now + window)
        return {0, 0, math.ceil(reset)}
    end
    """
    
    TOKEN_BUCKET_SCRIPT = """
    local key = KEYS[1]
    local rate = tonumber(ARGV[1])
    local capacity = tonumber(ARGV[2])
    local now = tonumber(ARGV[3])
    local requested = tonumber(ARGV[4])
    
    local data = redis.call('HMGET', key, 'tokens', 'last_update')
    local tokens = tonumber(data[1]) or capacity
    local last_update = tonumber(data[2]) or now
    
    local elapsed = now - last_update
    tokens = math.min(capacity, tokens + (elapsed * rate))
    
    local allowed = 0
    if tokens >= requested then
        tokens = tokens - requested
        allowed = 1
    end
    
    redis.call('HMSET', key, 'tokens', tokens, 'last_update', now)
    redis.call('EXPIRE', key, math.ceil(capacity / rate) * 2)
    
    local reset = now + ((capacity - tokens) / rate)
    return {allowed, math.floor(tokens), math.ceil(reset)}
    """
    
    def __init__(
        self,
        redis_client: redis.Redis,
        default_limit: int = 100,
        default_window: int = 60,
        algorithm: RateLimitAlgorithm = RateLimitAlgorithm.SLIDING_WINDOW,
        failure_mode: str = 'open',
        local_fallback_limit: int = 10
    ):
        self.redis = redis_client
        self.default_limit = default_limit
        self.default_window = default_window
        self.algorithm = algorithm
        self.failure_mode = failure_mode
        self.local_fallback_limit = local_fallback_limit
        
        # Register Lua scripts
        self.sliding_window_script = self.redis.register_script(self.SLIDING_WINDOW_SCRIPT)
        self.token_bucket_script = self.redis.register_script(self.TOKEN_BUCKET_SCRIPT)
        
        # Local fallback
        self.local_counts: dict[str, list[float]] = {}
        self.redis_healthy = True
    
    def is_allowed(
        self,
        key: str,
        limit: Optional[int] = None,
        window: Optional[int] = None,
        cost: int = 1
    ) -> RateLimitResult:
        """
        Check if request is allowed under rate limit.
        """
        limit = limit or self.default_limit
        window = window or self.default_window
        
        try:
            if self.algorithm == RateLimitAlgorithm.SLIDING_WINDOW:
                result = self._sliding_window_check(key, limit, window)
            else:
                result = self._token_bucket_check(key, limit, window, cost)
            
            self.redis_healthy = True
            return result
            
        except redis.RedisError as e:
            self.redis_healthy = False
            return self._fallback_check(key, limit, window)
    
    def _sliding_window_check(
        self,
        key: str,
        limit: int,
        window: int
    ) -> RateLimitResult:
        now = time.time()
        result = self.sliding_window_script(
            keys=[f"ratelimit:sw:{key}"],
            args=[now, window, limit]
        )
        
        return RateLimitResult(
            allowed=bool(result[0]),
            limit=limit,
            remaining=result[1],
            reset=result[2],
            retry_after=None if result[0] else max(0, result[2] - int(now))
        )
    
    def _token_bucket_check(
        self,
        key: str,
        limit: int,
        window: int,
        cost: int
    ) -> RateLimitResult:
        now = time.time()
        rate = limit / window
        capacity = limit
        
        result = self.token_bucket_script(
            keys=[f"ratelimit:tb:{key}"],
            args=[rate, capacity, now, cost]
        )
        
        return RateLimitResult(
            allowed=bool(result[0]),
            limit=limit,
            remaining=result[1],
            reset=result[2],
            retry_after=None if result[0] else max(0, result[2] - int(now))
        )
    
    def _fallback_check(self, key: str, limit: int, window: int) -> RateLimitResult:
        """
        Local fallback when Redis is unavailable.
        """
        if self.failure_mode == 'closed':
            return RateLimitResult(
                allowed=False,
                limit=limit,
                remaining=0,
                reset=int(time.time()) + window,
                retry_after=window
            )
        
        # Fail open with local limiting
        now = time.time()
        window_start = now - window
        
        if key not in self.local_counts:
            self.local_counts[key] = []
        
        # Remove old entries
        self.local_counts[key] = [t for t in self.local_counts[key] if t > window_start]
        
        count = len(self.local_counts[key])
        local_limit = min(self.local_fallback_limit, limit)
        
        if count < local_limit:
            self.local_counts[key].append(now)
            return RateLimitResult(
                allowed=True,
                limit=local_limit,
                remaining=local_limit - count - 1,
                reset=int(now + window)
            )
        else:
            return RateLimitResult(
                allowed=False,
                limit=local_limit,
                remaining=0,
                reset=int(now + window),
                retry_after=window
            )


# FastAPI integration
from fastapi import Request, HTTPException
from fastapi.responses import JSONResponse

def rate_limit_middleware(limiter: ProductionRateLimiter):
    async def middleware(request: Request, call_next):
        # Extract key from request
        key = request.headers.get('X-API-Key') or request.client.host
        
        result = limiter.is_allowed(key)
        
        if not result.allowed:
            return JSONResponse(
                status_code=429,
                content={
                    'error': 'rate_limit_exceeded',
                    'retry_after': result.retry_after
                },
                headers={
                    'X-RateLimit-Limit': str(result.limit),
                    'X-RateLimit-Remaining': str(result.remaining),
                    'X-RateLimit-Reset': str(result.reset),
                    'Retry-After': str(result.retry_after)
                }
            )
        
        response = await call_next(request)
        response.headers['X-RateLimit-Limit'] = str(result.limit)
        response.headers['X-RateLimit-Remaining'] = str(result.remaining)
        response.headers['X-RateLimit-Reset'] = str(result.reset)
        
        return response
    
    return middleware

A.2 Rate Limiter Testing Suite#

pythonTestSlidingWindowLimiter · 89 lines
import pytest
import time
import fakeredis
from unittest.mock import patch, MagicMock

class TestSlidingWindowLimiter:
    @pytest.fixture
    def limiter(self):
        redis_client = fakeredis.FakeRedis()
        return ProductionRateLimiter(
            redis_client=redis_client,
            default_limit=10,
            default_window=60,
            algorithm=RateLimitAlgorithm.SLIDING_WINDOW
        )
    
    def test_allows_under_limit(self, limiter):
        for i in range(10):
            result = limiter.is_allowed('user1')
            assert result.allowed, f"Request {i+1} should be allowed"
            assert result.remaining == 10 - i - 1
    
    def test_blocks_over_limit(self, limiter):
        for _ in range(10):
            limiter.is_allowed('user1')
        
        result = limiter.is_allowed('user1')
        assert not result.allowed
        assert result.remaining == 0
        assert result.retry_after > 0
    
    def test_separate_keys(self, limiter):
        for _ in range(10):
            limiter.is_allowed('user1')
        
        # Different key should still be allowed
        result = limiter.is_allowed('user2')
        assert result.allowed
    
    def test_window_reset(self, limiter):
        limiter.default_window = 1  # 1 second window
        
        for _ in range(10):
            limiter.is_allowed('user1')
        
        result = limiter.is_allowed('user1')
        assert not result.allowed
        
        time.sleep(1.1)
        
        result = limiter.is_allowed('user1')
        assert result.allowed


class TestFallbackBehavior:
    def test_fail_open(self):
        redis_client = MagicMock()
        redis_client.register_script.return_value = MagicMock(
            side_effect=redis.RedisError("Connection refused")
        )
        
        limiter = ProductionRateLimiter(
            redis_client=redis_client,
            failure_mode='open',
            local_fallback_limit=5
        )
        
        # Should use local fallback
        for i in range(5):
            result = limiter.is_allowed('user1')
            assert result.allowed
        
        # Local limit exceeded
        result = limiter.is_allowed('user1')
        assert not result.allowed
    
    def test_fail_closed(self):
        redis_client = MagicMock()
        redis_client.register_script.return_value = MagicMock(
            side_effect=redis.RedisError("Connection refused")
        )
        
        limiter = ProductionRateLimiter(
            redis_client=redis_client,
            failure_mode='closed'
        )
        
        result = limiter.is_allowed('user1')
        assert not result.allowed

this lesson in shorts · 3

play them all →

One idea from this lesson in about a minute each, made for your phone.

End of Day 03 · Rate Limiting at Scale
Finish Day 3Stages clear as you read them (0/5 so far). Skipping ahead doesn't count: marking complete unlocks after ~7 more min of reading.

Go deeper · reading trail

all reading →

Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.

StripeBlog post · ~9 min

Scaling your API with rate limiters

Paul Tarjan · 2017
What
The four limiters Stripe runs in front of its API, with sample code.
Key idea
Per-user request-rate and concurrency limiters (token buckets in Redis) sit alongside load shedders that drop low-priority traffic when the whole fleet is stressed.
Why here
Shows the distinction this lesson makes between rate limiting a client and load shedding for the system, and how to roll limiters out safely.
taught here: Token bucket
CloudflareBlog post · ~9 min

How we built rate limiting capable of scaling to millions of domains

Julien Desgats · 2017
What
How Cloudflare counts requests for rate limiting across its global edge.
Key idea
A sliding window approximated from the current and previous fixed-window counters needs only GET, SET and INCR, yet stays very close to an exact count.
Why here
The sliding window counter algorithm from this lesson, with the memory and accuracy reasoning behind choosing it at the edge.
taught here: Sliding window counter
FigmaBlog post · ~11 min

An alternative approach to rate limiting

Nikrad Mahdi · 2017
What
Figma compares token bucket, fixed window and sliding window log limiters before building its own.
Key idea
Its final design counts requests in small fixed windows, each 1/60th of the limit window, and sums them: a sliding window counter that is accurate but cheap in Redis memory.
Why here
A side-by-side comparison of the algorithms in this lesson, judged on the same two axes the lesson uses: accuracy and memory.
taught here: Rate limiting algorithms

New lessons by email

New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.

Discussion · …

Public: everyone reading this lesson sees comments. Be kind and keep it on the material.

Sign in to ask a question or answer one. Reading stays open to everyone.
loading the discussion…