Part 2: Infrastructure Building Blocks.

Cloudflare network (Example)

• North America: 55+ cities (LA, NYC, Chicago, Dallas, Seattle...)
• Europe: 55+ cities (London, Frankfurt, Paris, Amsterdam...)
• Asia Pacific: 50+ cities (Tokyo, Singapore, Sydney, Mumbai...)
• South America: 20+ cities (São Paulo, Buenos Aires...)
• Africa: 15+ cities (Johannesburg, Cape Town, Lagos...)
• Middle East: 10+ cities (Dubai, Tel Aviv...)

Total: 330+ cities across 100+ countries (as of 2025-26)

AWS CloudFront

• 750+ Points of Presence
• 15 Regional Edge Caches
• Integrated with AWS services

The closer the PoP to the user, the lower the latency.

• A user in Mumbai connecting to a Mumbai PoP = ~5-10ms
• Same user connecting to US origin = ~150-200ms

Long-term caching (Static assets with content hash)

Cache-Control: public, max-age=31536000, immutable

• public — Can be cached by CDN and browser
• max-age=31536000 — Cache for 1 year (365 * 24 * 60 * 60)
• immutable — Don't even check for updates (save revalidation)

Use for: app.a1b2c3.js, style.x7y8z9.css, image.abc123.png (Files with content hash in filename - changing content = new file)

Short-term caching (Semi-dynamic content)

Cache-Control: public, max-age=300, must-revalidate

• max-age=300 — Cache for 5 minutes
• must-revalidate — After TTL, MUST check with origin before serving

Use for: API responses, product listings, semi-dynamic pages

Revalidation caching (Always fresh, but allow caching)

Cache-Control: public, no-cache · ETag: "abc123"

• no-cache — DOESN'T mean "don't cache"!
• It means: Cache it, but revalidate with origin before using
• ETag allows conditional requests (If-None-Match: "abc123")
• If unchanged, origin returns 304 Not Modified (no body = fast)

Use for: Frequently updated content where staleness is unacceptable

No caching (Private/sensitive content)

Cache-Control: private, no-store

• private — Only browser can cache (not CDN)
• no-store — Don't cache at all, not even temporarily

Use for: User-specific data, authenticated responses, sensitive data

Different TTL for CDN vs browser

Cache-Control: public, max-age=3600, s-maxage=86400

• max-age=3600 — Browser caches for 1 hour
• s-maxage=86400 — CDN (shared cache) caches for 24 hours

Why? CDN can invalidate quickly. Browsers can't. So CDN can cache longer, browser should check more often.

Stale-while-revalidate (Performance optimization)

Cache-Control: public, max-age=300, stale-while-revalidate=86400

• For 5 minutes: Serve from cache
• For up to 24 hours after that: Serve stale content IMMEDIATELY, but fetch fresh content in background
• After that window: Must wait for fresh content

Benefit: User never waits for revalidation!

1. Versioned URLs (Recommended for static assets)

• Instead of: /static/app.js
• Use: /static/app.a1b2c3d4.js (content hash in filename)

How it works:

1. Build tool (Webpack, Vite) generates hash from file content
2. When code changes, hash changes, URL changes
3. New URL = automatic cache miss = users get fresh file
4. Old version stays cached (harmless, eventually expires)

Pros

• Instant updates for users
• Zero cache purge latency
• Safe rollbacks (old URLs still work)
• No CDN API calls needed

2. Purge API (For non-versioned content)

AWS CloudFront:

aws cloudfront create-invalidation \
  --distribution-id EDFDVBD632 \
  --paths "/images/*" "/api/products/*"

Cloudflare:

curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/purge_cache" \
  -H "Authorization: Bearer {token}" \
  -d '{"files":["https://example.com/image.jpg"]}'

Propagation time:

Pros

• Works for any URL pattern

Cons

• Propagation delay
• May have cost (CloudFront: 1000 free/month, then $0.005 each)

3. TTL-based expiry (Simple but has staleness window)

Cache-Control: max-age=300 (5 minutes)

• Content cached for 5 minutes
• After TTL, CDN checks with origin (If-Modified-Since)
• Origin returns 304 (unchanged) or 200 (new content)

Trade-off:

• Short TTL (60s) = More origin requests, always fresh
• Long TTL (1hr) = Fewer origin requests, may be stale

4. Cache tags / surrogate keys (Advanced)

Supported by: Fastly, Varnish, some CDNs

Response header: Surrogate-Key: product-123 category-shoes brand-nike

When product 123 updates: PURGE /tags/product-123

• → Invalidates ALL URLs tagged with "product-123"
• → Product page, category pages, search results, recommendations...

Pros

• Granular invalidation without knowing all URLs
• One purge affects all related content

Cons

• Requires CDN support

Best practice: Combine strategies

CloudFront (AWS)

Network: 750+ edge locations, 15 regional edge caches

Strengths

• Deep AWS integration (S3, ALB, API Gateway, Lambda@Edge)
• Lambda@Edge for compute at edge (auth, redirects, A/B testing)
• Origin Shield (extra caching layer to protect origin)
• Real-time logs to Kinesis
• Field-level encryption
• WebSocket support

Weaknesses

• Slower invalidation (usually minutes, not milliseconds)
• Complex pricing model
• Configuration can be verbose

Best for: AWS-native applications, serverless architectures

Pricing: ~$0.085/GB (varies by region), $0.0075/10K HTTP requests ($0.01/10K HTTPS)

Cloudflare

Network: 330+ cities worldwide

Strengths

• Generous FREE tier (yes, really free)
• Built-in DDoS protection (industry-leading)
• Workers (JavaScript/WASM at edge - very powerful)
• Workers KV (distributed key-value at edge)
• Instant cache purge (<150ms global average)
• Easy setup (just change DNS nameservers)
• Fast public DNS resolver (1.1.1.1, often top-ranked in benchmarks)
• Argo Smart Routing (optimized paths)

Weaknesses

• Less control over caching behavior than some competitors
• Advanced features require paid plans
• Must use Cloudflare DNS on Free/Pro plans (CNAME setup needs Business+; can be a dealbreaker)

Best for: Startups, security-focused apps, edge computing

Pricing: Free tier, Pro $20/mo, Business $200/mo (billed annually; $25 and $250 billed monthly), Enterprise custom

Fastly

Network: 100+ PoPs (fewer but very high capacity)

Strengths

• INSTANT PURGE (<150ms global) - their killer feature
• VCL (Varnish Configuration Language) for custom caching logic
• Real-time analytics and logging
• Fastly Compute (formerly Compute@Edge; WebAssembly at edge)
• Image optimization built-in
• Surrogate keys for granular invalidation

Weaknesses

• Fewer PoPs than competitors (100+ vs 330+)
• Steeper learning curve (VCL is powerful but complex)
• No free tier (only a $50 developer trial credit)
• Higher cost for small sites

Best for: Media streaming, real-time content, advanced caching needs

Pricing: $0.12/GB + $0.0075/10K requests (minimum ~$50/mo)

Akamai

Network: 4,000+ locations (largest network by far)

Strengths

• Largest, most distributed network in the world
• Enterprise-grade SLAs (99.99%+)
• Advanced security (WAF, Bot Management, DDoS)
• Media delivery specialization (used by major streaming services )
• API acceleration

Weaknesses

• Expensive (enterprise pricing)
• Complex configuration and management
• Sales-driven (hard to get started without talking to sales)

Best for: Large enterprises, media companies, global reach needs

Pricing: Enterprise contracts (typically $$$$$)

Quick selection guide

1. Round robin

How it works:

• Request 1 → Server A
• Request 2 → Server B
• Request 3 → Server C
• Request 4 → Server A (cycle repeats)
• Request 5 → Server B
• ...

Implementation: next = servers[counter % len(servers)]; counter++

Pros

• Simple to implement and understand
• Even distribution over time
• No state to maintain
• Works well with homogeneous servers

Cons

• Ignores server capacity (all treated equal)
• Ignores current load
• Some requests may take longer than others

Best for: Identical servers, stateless requests, similar workloads

2. Weighted round robin

Pros

• Accounts for different server capacities
• Useful during migrations (old=1, new=5)
• Canary deployments (stable=99, canary=1)

Cons

• Weights are static, don't adapt to actual load
• Requires manual tuning

Best for: Heterogeneous servers, gradual rollouts, canary deploys

3. Least connections

Current state:

• Server A: 150 active connections
• Server B: 75 active connections ← Next request goes here
• Server C: 120 active connections

How it works:

1. LB tracks active connections per server
2. New request → server with fewest connections
3. Connection count incremented when request starts
4. Connection count decremented when response completes

Pros

• Automatically adapts to actual load
• Handles varying request durations well
• Self-balancing

Cons

• Requires connection tracking (slight overhead)
• Doesn't account for connection "weight" (some heavier than others)

Best for: Long-running connections, WebSockets, variable workloads

4. Least response time

Current metrics:

• Server A: avg response = 45ms, 50 connections
• Server B: avg response = 28ms, 60 connections ← Fastest!
• Server C: avg response = 52ms, 40 connections

Calculation: Route to server with lowest (response_time * connections)

Pros

• Optimizes for actual user-perceived latency
• Automatically adapts to server performance
• Accounts for both load and speed

Cons

• More complex to implement
• Can be "noisy" with variable latencies
• May starve consistently slow servers

Best for: Latency-sensitive applications, heterogeneous performance

5. IP hash (Source IP Affinity)

How it works: server_index = hash(client_ip) % number_of_servers

Same client always goes to same server (until server pool changes)

Pros

• Sticky sessions without cookies
• Good for local server-side caching
• Stateless LB (no session table needed)

Cons

• Can cause uneven distribution
• Adding/removing server reshuffles many clients
• Mobile users may change IP (WiFi → cellular)
• Users behind NAT share IP

Best for: Caching layers, when cookies aren't available

6. Consistent hashing (Advanced)

How it works:

• Servers placed on a virtual ring (hash space 0 to 2^32 - 1)
• Request key hashed to point on ring
• Route to first server clockwise from that point
• On the ring: Server A → Server B → Server C (clockwise)

Key benefit: Adding/removing server only affects 1/N of keys! (Unlike IP hash where many clients get reshuffled)

Best for: Distributed caches, databases, when servers change often

Algorithm selection guide

Default choice: Least Connections (adapts well to most scenarios)

OSI model reference: Layer 4 = Transport Layer (TCP/UDP) · Layer 7 = Application Layer (HTTP/HTTPS)

Layer 4 (L4)

What it sees — TCP/IP Packet Headers:

• Source IP: 192.168.1.100
• Destination IP: 10.0.0.1
• Source Port: 52431
• Destination Port: 443
• Protocol: TCP
• [Payload is encrypted/opaque - LB cannot read it]

What it can do

• Route based on IP address and port
• Very fast (no packet inspection, just forward)
• Protocol agnostic (TCP, UDP, any port)
• Preserve client IP (DSR/Direct Server Return mode)
• Handle millions of connections

What it cannot do

• Route based on URL path (/api vs /static)
• Read HTTP headers or cookies
• Modify request/response content
• Smart routing based on content
• SSL termination (usually passed through; AWS NLB can terminate TLS)

Use cases: Database connections (MySQL, PostgreSQL) · Redis, Memcached · Gaming servers (UDP) · Any non-HTTP TCP/UDP service · When you need maximum performance

AWS: Network Load Balancer (NLB) · GCP: TCP/UDP Load Balancer

Performance: ~1M+ connections/sec, <100μs added latency

Layer 7 (L7)

What it sees (after SSL termination) — Full HTTP Request:

• Method: GET
• Path: /api/v2/users/123
• Host: api.example.com
• Headers:
  • Authorization: Bearer eyJhbGc...
  • Content-Type: application/json
  • X-Request-ID: abc123
  • X-Feature-Flag: new-checkout
• Cookies:
  • session_id=xyz789
• Body: {"name": "John"}

What it can do

• Route based on URL path (/api/* → API servers)
• Route based on hostname (api.* vs www.*)
• Route based on HTTP headers
• Route based on cookies (session affinity)
• Route based on query parameters
• SSL/TLS termination
• Add/modify/remove headers
• URL rewriting (/old-path → /new-path)
• Response caching
• Compression (gzip)
• Rate limiting
• Authentication
• WebSocket support

Example L7 routing rules:

Use cases: Web applications · REST APIs · Microservices routing · A/B testing (route 10% to new version) · Canary deployments · Multi-tenant applications (route by subdomain) · Blue-green deployments

AWS: Application Load Balancer (ALB) · GCP: HTTP(S) Load Balancer

Performance: ~100K connections/sec, 1-5ms added latency

Comparison summary

When to use which:

• HTTP/HTTPS traffic → L7 (ALB)
• Database connections → L4 (NLB)
• Need content-based routing → L7
• Need maximum throughput → L4
• Non-HTTP protocols → L4

Purpose: Detect unhealthy servers and stop routing traffic to them

Configuration example (AWS ALB)

Health check timeline example

Types of health checks

1. Shallow / liveness check — Just verifies the process is running and can respond

GET /health Response:

{
  "status": "ok"
}

Pros

• Fast, low overhead
• Always succeeds if process is up
• Good for load balancer health checks

Cons

• Doesn't verify dependencies work

2. Deep / readiness check — Verifies the app AND all its dependencies are healthy

GET /health/ready Response (healthy):

{
  "status": "healthy",
  "version": "1.2.3",
  "uptime_seconds": 86400,
  "checks": {
    "database": {
      "status": "up",
      "latency_ms": 5
    },
    "redis": {
      "status": "up",
      "latency_ms": 1
    },
    "external_api": {
      "status": "up",
      "latency_ms": 45
    }
  }
}

Response (unhealthy - HTTP 503):

{
  "status": "unhealthy",
  "checks": {
    "database": {
      "status": "down",
      "error": "Connection refused"
    },
    "redis": { "status": "up" },
    "external_api": { "status": "up" }
  }
}

Pros

• Comprehensive status
• Catches dependency failures

Cons

• Slower (queries dependencies)
• Risk: If DB is down, ALL servers marked unhealthy!

Best practice

Use SHALLOW check for load balancer (keep traffic flowing). Use DEEP check for monitoring/alerting (know what's broken).

Cloud load balancers

AWS

• ALB (Application Load Balancer) - Layer 7
  • HTTP/HTTPS traffic
  • Path-based and host-based routing
  • WebSocket and HTTP/2 support
  • Integrated with WAF, Cognito
  • ~$0.0225/hour + $0.008/LCU
• NLB (Network Load Balancer) - Layer 4
  • TCP/UDP/TLS traffic
  • Ultra-low latency (<100μs)
  • Millions of requests per second
  • Static IP / Elastic IP support
  • ~$0.0225/hour + $0.006/NLCU

GCP

• HTTP(S) Load Balancer - Global L7
  • Single anycast IP worldwide
  • Automatic multi-region failover
  • Cloud CDN integration
• Network Load Balancer - Regional L4
  • High-performance TCP/UDP
  • Regional scope

Software load balancers

NGINX

• Most popular web server / reverse proxy
• HTTP, TCP, UDP load balancing
• Open source (free) + NGINX Plus (commercial)
• Event-driven, high performance
• Great documentation and community

Config example:

upstream backend {
    least_conn;
    server 10.0.0.1:8080 weight=3;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080 backup;
}
server {
    listen 80;
    location / {
        proxy_pass http://backend;
    }
}

HAProxy

• High-performance TCP/HTTP load balancer
• Battle-tested (GitHub, Reddit, Stack Overflow, Twitter)
• Very detailed metrics and logging
• Advanced health checking

Envoy

• Modern, cloud-native proxy
• Originally from Lyft, now CNCF
• Foundation of service meshes (Istio, AWS App Mesh)
• Native gRPC support
• Dynamic configuration via xDS API
• Best for: Kubernetes, service mesh, microservices

Traefik

• Kubernetes-native
• Auto-discovery of services
• Automatic Let's Encrypt certificates
• Best for: Kubernetes, Docker Swarm

A record (Address)

Maps domain to IPv4 address

example.com.      300   IN   A   93.184.216.34
api.example.com.  60    IN   A   10.0.1.100

Multiple A records = DNS round-robin load balancing

api.example.com.  60   IN   A   10.0.1.100
api.example.com.  60   IN   A   10.0.1.101
api.example.com.  60   IN   A   10.0.1.102

AAAA record (IPv6)

Maps domain to IPv6 address

example.com.  300  IN  AAAA  2606:2800:220:1:248:1893:25c8:1946

CNAME record (Canonical Name / Alias)

Creates an alias pointing to another domain

www.example.com.   300  IN  CNAME  example.com.
cdn.example.com.   300  IN  CNAME  d123456.cloudfront.net.
blog.example.com.  300  IN  CNAME  mycompany.ghost.io.

Important: Cannot use CNAME at zone apex (example.com). Workarounds: ALIAS (Route 53), ANAME, CNAME flattening (Cloudflare)

MX record (Mail Exchange)

Specifies mail servers for the domain

example.com.  300  IN  MX  10  mail1.example.com.
example.com.  300  IN  MX  20  mail2.example.com.  (backup)

The number after MX is the Priority (lower = higher priority).

TXT record (Text)

Stores text, used for verification and email security

; Domain verification:
example.com.  IN  TXT  "google-site-verification=abc123..."
; SPF (email authentication):
example.com.  IN  TXT  "v=spf1 include:_spf.google.com ~all"
; DMARC (email policy):
_dmarc.example.com.  IN  TXT  "v=DMARC1; p=reject; rua=..."

NS record (Name Server)

Delegates DNS authority to nameservers

example.com.  86400  IN  NS  ns1.example.com.
example.com.  86400  IN  NS  ns2.example.com.

1. Geographic routing (GeoDNS)

Route users to nearest datacenter:

Benefits: Lower latency, data residency compliance

2. Failover

Automatic switch to backup when primary fails:

• Primary healthy: api.example.com → primary-dc.example.com
• Primary fails: api.example.com → backup-dc.example.com

TTL affects failover speed! Lower TTL = faster failover

3. Weighted routing (Canary Deployments)

Split traffic between versions — api.example.com:

• Weight 95: stable.internal (v1.0)
• Weight 5: canary.internal (v1.1)

4. Service discovery (Internal DNS)

Services find each other by DNS name:

• user-service.internal → 10.0.1.1, 10.0.1.2, 10.0.1.3
• order-service.internal → 10.0.2.1, 10.0.2.2

No hardcoded IPs! Works with auto-scaling.

SQL (Relational) databases

Examples: PostgreSQL, MySQL, SQL Server

Structure: Fixed schema, tables with rows and columns

ACID transactions:

BEGIN;
  UPDATE accounts SET balance = balance - 100 WHERE id = 1;
  UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;  -- Both happen or neither happens

Best for

• Complex queries with JOINs
• Transactions (banking, orders)
• Data integrity critical
• Structured, predictable data
• Reporting and analytics

Default choice: PostgreSQL (handles most use cases well)

NoSQL database types

1. Document store (MongoDB, CouchDB)

{
  "_id": "user_123",
  "name": "John",
  "orders": [                    // Embedded documents
    {"id": "ord1", "total": 99.99}
  ]
}

• ok: Flexible schema, natural for hierarchical data
• no: No JOINs (denormalize instead)
• Best for: Content management, catalogs, user profiles

2. Key-value store (Redis, DynamoDB)

• user:123 → {"name": "John", "email": "j@ex.com"}

• session:abc → {"user_id": 123, "expires": ...}

• ok: Extremely fast O(1) lookups

• no: No complex queries

• Best for: Sessions, caching, real-time data, leaderboards

3. Wide-column store (Cassandra, HBase)

• ok: Massive write throughput, linear scaling
• no: Limited query patterns
• Best for: Time-series, IoT, logging, high write volume

4. Graph database (Neo4j, Neptune)

• (Alice) —FOLLOWS→ (Bob) —WORKS_AT→ (Acme)

• ok: Natural for connected data, fast traversals

• Best for: Social networks, recommendations, fraud detection

Database selection quick guide

Why caching matters

Without cache: Every request → Database (50ms). 1000 requests = 1000 DB queries = DB overload

With cache (90% hit rate):

• 900 requests → Cache (1ms)
• 100 requests → Database (50ms)
• Average latency: (0.9 × 1) + (0.1 × 50) = 5.9ms
• vs 50ms without cache = 8.5x faster!
• Database load reduced by 90%!

Cache-aside pattern (Most Common)

def get_user(user_id):
    # 1. Check cache first
    cached = redis.get(f"user:{user_id}")
    if cached:
        return json.loads(cached)  # Cache HIT

    # 2. Cache MISS - query database
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)

    # 3. Store in cache for next time
    redis.setex(f"user:{user_id}", 3600, json.dumps(user))  # 1hr TTL

    return user

# INVALIDATION (on update):
def update_user(user_id, data):
    db.update(user_id, data)
    redis.delete(f"user:{user_id}")  # Invalidate cache

Cache eviction policies

Redis vs Memcached

• Redis: Rich data structures, persistence, pub/sub, Lua scripts
• Memcached: Simpler, multi-threaded, more memory efficient

Default choice: Redis (more versatile)

Structure

Bucket: my-app-uploads

• images/profiles/user123.jpg (Object)
• videos/uploads/intro.mp4 (Object)
• documents/reports/q3.pdf (Object)

Characteristics

Pros

• Virtually unlimited scale
• 99.999999999% durability (11 nines!)
• HTTP access (easy CDN integration)
• Cheap (compared to block storage)
• Versioning, lifecycle policies

Cons

• Not a native filesystem (FUSE tools like Mountpoint/s3fs exist, with limits)
• No partial updates (replace whole object)
• Higher latency than local disk

Use cases

• User uploads (images, videos, documents)
• Static website assets
• Backups and archives
• Data lake storage

Providers

• AWS S3 (the original, most popular)
• Google Cloud Storage
• Azure Blob Storage
• Cloudflare R2 (S3-compatible, no egress fees!)

Default choices:

• Database: PostgreSQL
• Cache: Redis
• Queue: SQS (AWS) or Kafka (high throughput)
• CDN: CloudFront (AWS) or Cloudflare
• Object Storage: S3

Edge layer

• CDN: Cache static content at edge, 95%+ hit ratio target
• Load Balancer: Distribute traffic, L4 vs L7, health checks
• API Gateway: Auth, rate limiting, routing, transformation
• DNS: Domain resolution, geo routing, failover

Application layer

• Keep servers STATELESS (externalize state to Redis/DB)
• Start MONOLITH, extract microservices when needed

Data layer

• SQL (PostgreSQL): Transactions, relationships, complex queries
• NoSQL: Document (MongoDB), Key-Value (Redis), Column (Cassandra)
• Caching: Cache-aside pattern, LRU eviction
• Object Storage: S3 for files

Messaging layer

• Queues: Async processing, decoupling, traffic buffering
• Kafka: High-throughput event streaming
• Event-driven: Loose coupling, failure isolation