Want it harder? Hide the model answer and work each phase in a scratchpad against its time budget, then reveal and score yourself. Or simply read on: the checkpoints below work either way.
How Do You Sell 500 Seats to 2 Million People in 60 Seconds — Without Selling One Seat Twice?
It is 9:59:58 AM. Across India, in trains, buses, cyber cafés, offices, and bedrooms, two million fingers hover over the "Search" button. They are all trying to do the same thing: book a Tatkal ticket on the same popular train, for the same date, in the same class.
At 10:00:00 AM, the gates open. Ten seconds later, every seat is held, and within about five minutes, once the last payments finish, the train is sold out. Ninety-nine point nine seven five percent of the people who tried will fail — and they need to fail cleanly, instantly, and fairly. Not one seat can be sold twice. Not one rupee can be double-charged. And the 0.025% who succeed need their booking, and only their booking, confirmed.
Some version of this plays out every morning across India. This is the system we'll design today.
- 14.5 Lakh (1.45M) — average tickets/day (FY 2025-26)
- 18.4 Lakh (1.84M) — record single day (19 Aug 2025)
- 37,410 — record tickets booked in one minute (16 Aug 2025)
- 1.5 Lakh+ bookings/min — booking capacity of the revamped site (live 15 Jul 2026), up from ~32,000/min
- 40 Lakh+ enquiries/min — enquiry capacity after the same upgrade, up from ~4 lakh/min: reads dwarf writes
- 10:00 AM / 11:00 AM — Tatkal windows: AC classes 10:00 AM, Non-AC classes 11:00 AM
- 3.03 Crore (30.3M) — suspicious user IDs deactivated in the 2025 clean-up drive
- 4.18 Lakh (418,000) — suspicious PNRs linked to complaints on the National Cyber Crime Portal
For this design we'll assume:
One popular train's Tatkal quota is ~500 seats
On a hot route, that quota attracts a couple of million hopeful clicks
The system must reject 99%+ of requests in milliseconds, not seconds
One wrong seat allocation means a real family stranded at a real station
This is the system we'll design today.
At 10:00 about 2 million people try for one train's ~500 Tatkal seats, ~280,000 requests a second in the first 5 seconds, so this is admission control, not storage. A cached read plane serves search; a waiting room, an atomic counter per pool and a single-writer allocator let only the admitted hold, pay and get a PNR. The deep dives cover the hot row, adaptive admission, the booking saga and the defences against bots and touts, then the 10:00 playbook, late payments, double taps and fifty hot trains at once.
The Interview Begins#
You're interviewing at a company building ticketing infrastructure for a national transit agency. The principal architect leans forward:
"Everyone assumes booking a train ticket is 'just a database write.' It's not. I want you to design India's Tatkal ticket booking system — the part of Indian Railways' IRCTC platform that opens exactly 500 seats to millions of people at a precise second, every single morning."
At exactly 10:00:00 AM, open booking for a Tatkal quota of ~500 seats on a single popular train. Assume roughly 2 million people will attempt to book within the first minute.
Requirements
- Zero double-booking — a seat is sold to exactly one passenger
- Zero double-charging — a payment is captured exactly once
- Fair, fast rejection for the ~99.975% who don't get a seat
- End-to-end booking (search → seat → pay → PNR) in seconds
- Resilient to script-based bots and bulk-booking agents
- The other 12,999 trains running that day must be unaffected
"Most candidates think this is a database problem. It's actually an admission-control problem wearing a database's clothes. Show me you understand the difference."
Phase 1: Requirements Clarification (5 minutes)#
"Before I design anything, I want to understand exactly where the contention lives."
Your Questions#
"First — is the entire IRCTC platform this contended, or is it specific trains?"
"Great distinction. Most of IRCTC's ~14.5 lakh tickets a day are boring — general quota, booked hours or days in advance, spread evenly across ~13,000 trains. The chaos is concentrated: a handful of popular routes (say, a Rajdhani or a Vande Bharat between two big cities) have a Tatkal quota — a small block of seats released one day before the journey date, at a fixed time (10 AM for AC classes, 11 AM for non-AC). That's where, in our scenario, 2 million people converge on 500 seats."
"So the seat inventory itself — how is it structured? Is it one row per seat, or something else?"
"Think of inventory as a pool: one bucket per (train, date, class, quota). A given train running tomorrow, in 3AC, under the Tatkal quota, might have exactly 486 seats in its pool. Every booking is really an allocation from a pool, followed by assigning a specific berth within it."
"What happens if a payment succeeds but the app crashes before showing the PNR? Or if a user's card is charged but the seat was already taken by someone faster?"
"Both must be handled with zero ambiguity. If we can't confirm a seat, the money must come back automatically — no support ticket required. And a user should never see 'payment successful' for a seat that isn't theirs."
"How do you stop travel agents and bots from buying up the Tatkal quota in the first few seconds and reselling tickets?"
"This is a real, ongoing arms race. As of mid-2025, Indian Railways mandated Aadhaar-based OTP authentication for Tatkal bookings and banned authorized agents from booking in the first 30 minutes of the window. Your design should assume adversarial traffic, not just heavy traffic."
"Last one — is search/availability-checking the same code path as booking?"
"Absolutely not, and that's a trap many candidates fall into. Enquiries dwarf bookings: IRCTC's July 2026 upgrade sized the site for 40 lakh+ enquiries a minute against 1.5 lakh+ bookings a minute (before it, ~4 lakh against ~32,000). If your search path and your booking path share infrastructure, the search traffic alone will take down bookings."
Functional Requirements#
1. Search & availability
- Search trains by source, destination, date, class
- Show live-ish seat/berth availability per quota
- Show fare, route, duration
2. Booking
- Reserve seats from a (train, date, class, quota) pool
- Collect passenger details (name, age, gender, berth preference)
- Aadhaar-OTP-based identity verification for Tatkal
- Capture payment (UPI, card, net banking, wallet)
- Generate a PNR (Passenger Name Record) on confirmation
3. Inventory states
- CNF (Confirmed) — allocated a real berth
- RAC (Reservation Against Cancellation) — shares a berth, guaranteed seat
- WL (Waitlisted) — ordered position, no seat yet
4. Post-booking
- Cancellation with tiered refund rules
- Automatic promotion: WL → RAC → CNF as seats free up
- First chart prepared well before departure: by 8 PM the previous day for trains leaving 5 AM–2 PM, at least ~10 hours ahead for the rest (since Dec 2025; it was ~4 hours until mid-2025)
- SMS / push notification for status changes
5. Anti-abuse
- One Tatkal booking per authenticated user per opening window
- Agent booking restriction in the first 30 minutes
- Bot / scripted-traffic detection
Non-Functional Requirements#
Scale (for one hot train's Tatkal window)
- ~2,000,000 booking attempts within 60 seconds
- ~500 seats available to allocate
- Contention ratio: 4,000 requests for every 1 seat
Scale (platform-wide, real IRCTC figures)
- 14.5 lakh tickets/day average (FY 2025-26), 18.4 lakh on the record day (19 Aug 2025)
- 37,410 tickets booked in a single minute (record, 16 Aug 2025)
- Capacity after the July 2026 upgrade: 40 lakh+ enquiries/minute, 1.5 lakh+ bookings/minute (was ~4 lakh and ~32,000)
Latency
- Admission decision (in queue / rejected): < 200ms
- Seat hold confirmation: < 1 second
- End-to-end booking (search already done → PNR): < 15 seconds
Consistency
- Zero oversell: never allocate the same berth twice
- Exactly-once payment capture per booking attempt
- Every debited rupee is either tied to a confirmed PNR or refunded
Availability
- A single hot train must never degrade booking for other trains
- Search/enquiry traffic must never degrade the booking path
Security
- Identity-verified booking (Aadhaar OTP) for Tatkal
- Detect and throttle scripted / bot traffic in real time
- Rate-limit and flag bulk-booking patterns
Phase 2: Back of the Envelope Estimation (5 minutes)#
"Let me size this precisely — because the numbers reveal why this is an admission-control problem, not a storage problem."
Before the numbers: where does the load really land?
Two million attempts arrive in the first minute, but about 70% of them land in the first five seconds. What is the peak request rate? And how much durable storage do the 1,999,500 losing attempts need?
One strong answer
- Smooth rate: 2,000,000 ÷ 60 s ≈ 33,000 requests/s, but it isn't smooth.
- Real peak: 1,400,000 ÷ 5 s = ~280,000 requests/s, for one train. Even on its record minute, the whole national platform booked about 620 tickets a second.
- Storage: the 500 winners need roughly 350 KB. The losers need zero bytes: a rejection must never touch durable storage.
So this is an admission-control problem, not a storage problem. The job is to say no to almost everyone as early and cheaply as possible.
Traffic Estimation#
- Booking attempts (60-second window): 2,000,000
- Available seats: 500
- Contention ratio: 4,000 : 1
- If arrivals were perfectly smooth over 60s — average request rate: ~33,300 req/sec
- Reality — Tatkal traffic is NOT smooth. Most of it lands in the first 2-5 seconds, because users are refreshing a page and hitting the same clock. Requests in first 5 seconds (~70% of total): 1,400,000
- Effective peak rate: ~280,000 req/sec
- For comparison, IRCTC's ENTIRE PLATFORM record (all trains, nationally) is 37,410 bookings in one minute = ~620 bookings/sec. One hot train's opening burst would be ~450x that if left unthrottled.
- Per booking attempt, multiple backend operations:
- Session / login validation: 1 lookup
- Availability read: 1-2 reads
- Seat hold attempt: 1 write (mostly rejected)
- Passenger form + Aadhaar OTP: 1-2 calls (only for winners)
- Payment initiate + webhook: 2 calls (only for winners)
- PNR write + SMS: 1-2 writes (only for winners)
- Effective backend ops/sec at peak: ~300,000+ for this ONE train
Storage Estimation#
- Per PNR record:
- PNR number: 8 bytes
- Train, date, class, quota: ~40 bytes
- Passenger details (×1-6): ~300 bytes
- Berth allocations: ~60 bytes
- Payment reference: ~40 bytes
- Timestamps, status: ~40 bytes
- Total: ~500-700 bytes
- 500 successful bookings for this train: ~350 KB
- 1,999,500 failed/rejected attempts: 0 bytes stored (rejections should never touch durable storage)
- Platform-wide, daily:
- ~1.45M confirmed PNRs × 700B: ~1 GB/day
- Waitlist + cancellation history: ~2-3 GB/day
- Enquiry/search logs (analytics tier): tens of GB/day
- 7-year audit retention (bookings only): ~2.5 TB
Conclusion: The bottleneck is never disk. It is the number of requests that must be evaluated, rejected, or serialized against 500 rows in the same second.
Key Metrics Summary#
| Metric | Value |
|---|---|
| Requests for this train (60s) | 2,000,000 |
| Seats available | 500 |
| Success rate | 0.025% |
| Peak instantaneous rate | ~280,000 req/sec |
| Storage for winners | ~350 KB |
| Storage for losers | 0 bytes (rejected in flight) |
| National record: tickets/min | 37,410 |
| National capacity, enquiries vs bookings/min (2026) | 40 lakh+ vs 1.5 lakh+ |
Requirements and estimation
Q1/3 2,000,000 attempts arrive for one hot train, and about 70% of them land in the first 5 seconds. What peak rate must the front door survive?
Answer & explanation
Answer: C
1.4M requests / 5 s = ~280,000 req/s. The 33,000/s figure assumes smooth arrivals, which Tatkal traffic never has; ~620/s is the whole platform's record booking rate (37,410 in one minute).
Q2/3 The estimate concludes "storage is trivial". What is the real bottleneck?
Answer & explanation
Answer: B
Winners need ~350 KB in total and rejected attempts should store nothing. The hard part is deciding fast for millions of callers, not persisting data.
Q3/3 A user's money is debited but the seat cannot be confirmed. What do the requirements demand?
Answer & explanation
Answer: D
Every debited rupee must be tied to a confirmed PNR or refunded. Making the user chase a refund breaks that guarantee.
Phase 3: High-Level Design (10 minutes)#
Sketch it first: should search and booking share anything?
IRCTC sizes its site for far more enquiries than bookings (40 lakh+ vs 1.5 lakh+ a minute after the 2026 upgrade). Would you let searching and booking share servers, caches or databases? Sketch the parts a booking passes through.
One strong answer
No. Their failure modes are opposite, so they get physically separate planes:
- Read plane (search and availability): caches and replicas, scaled out, aggressively rate-limited, allowed to be a second stale.
- Write plane (booking): a virtual waiting room at the edge → an atomic admission counter per (train, date, class, quota) pool → a single-writer seat allocator → a hold-and-pay saga with the payment gateway → the PNR.
The waiting room's only job is to protect the write plane from the read plane's volume, so a search spike can never take booking down, and booking can never be slowed by people refreshing availability.
"The core insight: this system needs two completely separate planes — a cheap, cacheable, horizontally-scalable read plane for search/enquiry, and a tiny, tightly-serialized write plane for the actual seat allocation. Mixing them is how real ticketing platforms fall over."
System Architecture#
%% title: IRCTC platform architecture
flowchart TD
subgraph L1["Layer 1 · Client surfaces"]
direction TB
WEB@{ icon: "sd:browser", label: "Website" }
APP@{ icon: "sd:phone", label: "Rail Connect (App)" }
PRS@{ icon: "sd:ticket", label: "PRS Counter (offline)" }
AGT@{ icon: "sd:desktop", label: "Agent Terminals" }
end
subgraph L2["Layer 2 · Edge / admission control"]
RD["READ PLANE (cheap)<br/>CDN + cache for search, fares,<br/>availability snapshots (refreshed every 1-2 sec)<br/>Absorbs lakhs of enquiries/min here"]:::cache
WR[["VIRTUAL WAITING ROOM<br/>Issues queue tokens, admits users at a rate<br/>the write plane can actually absorb.<br/>Rejects/queues the ~99.975% who can't win."]]:::queue
end
subgraph L3["Layer 3 · Booking core (write plane)"]
SI["Session & Identity<br/>(Aadhaar OTP, device binding)"]:::service
INV["Seat Inventory Service<br/>One pool per (train, date, class, quota).<br/>Single-writer per pool = no oversell"]:::hot
SAGA["Booking Saga Orchestrator<br/>Hold → passenger details → payment → PNR,<br/>or compensate (release + refund)"]:::core
FR["Fraud / Risk Engine<br/>Bot detection, velocity checks,<br/>agent-window restriction"]:::service
SI & INV & SAGA --> FR
end
subgraph L4["Layer 4 · Settlement & downstream"]
direction TB
PG@{ icon: "sd:card", label: "Payment Gateways<br/>(UPI, cards, netbanking)" }
PNR@{ icon: "sd:database", label: "PNR / Ticketing DB<br/>(active-passive)" }
WL["Waitlist/RAC Promotion<br/>& Chart Prep"]:::service
NT@{ icon: "sd:sms", label: "SMS / Push Notify" }
end
L1 ==> L2
WR ==> L3
FR ==> L4
class WEB,APP,PRS,AGT client
class PG,NT external
class PNR storeKEY INSIGHT:
- The Virtual Waiting Room's ONLY job is to protect Layer 3 from Layer 1's volume
- The Seat Inventory Service for a hot pool is intentionally NOT horizontally scaled — it is a single, fast, serialized writer, because correctness for 500 rows matters infinitely more than throughput for 500 rows
Data Flow: Booking a Tatkal Ticket#
"Let's trace Priya, who wants seat #501 of 500 possible winners..."
%% title: Tatkal booking flow — Priya vs 1,999,999 others sequenceDiagram autonumber participant A as Priya's App participant W as Waiting Room participant B as Booking Core participant P as Payment/PNR Note over A: 09:59:50 · Pre-fetches search results, keeps session open A->>W: 10:00:00 · "Request a queue token" W-->>A: "Token issued — position 1,842, estimated wait 2 seconds" W->>B: "Admitted — rate set by whatever Booking Core can absorb right now" A->>B: "Select seat + hold" Note over A,B: reply: "3AC, pool has 12 seats left" B-->>A: "Hold granted (TTL 5 minutes)" A->>B: "Passenger details + Aadhaar OTP" A->>P: "Initiate payment" P-->>A: "Payment success" B-->>A: "PNR confirmed: 2847193651 — Seat 3A, Berth 24" Note over A,P: TOTAL TIME: ~8-12 seconds for a winner
MEANWHILE, IN PARALLEL:
- 1,999,500 other requests are rejected at STEP 2 or STEP 3 in < 200ms
- They never reach the Seat Inventory Service at all
- This is the whole point: fail cheap, fail fast, fail at the edge
High-level design
Q1/3 Why is the Seat Inventory Service for a hot pool deliberately NOT scaled horizontally?
Answer & explanation
Answer: A
Multiple writers over the same 500 seats would need coordination to avoid double allocation. The waiting room keeps load low enough that one fast writer suffices.
Q2/3 Priya's app shows "12 seats left" from the read plane, but her hold request is rejected as sold out. Is this a bug?
Answer & explanation
Answer: C
The read plane trades freshness for absorbing lakhs of enquiries a minute cheaply. A stale count hurts no one; a stale booking decision would oversell.
Q3/3 Where should the ~1,999,500 attempts that cannot win be turned away?
Answer & explanation
Answer: D
Fail cheap, fail fast, fail at the edge. Letting losers reach the inventory service would recreate the hot-row contention the design avoids.
Phase 4: Deep Dives (20 minutes)#
Deep Dive 1: 500 Seats, Millions of Hands — The Hottest Rows in the Database (Week 1 Concepts: Hot Keys & Skew)#
"The single biggest risk in this design isn't the database being too small — it's 500 rows becoming, for 60 seconds, the hottest data in the entire country's transactional systems."
500 rows, 280,000 hands
Why does a plain UPDATE pool SET available = available - 1 WHERE available > 0 fall over at 280,000 requests a second? Why doesn't splitting the pool into ten rows of 50 seats fix it? What would you put in front of the database?
One strong answer
- The UPDATE: every request queues for the same row lock, so the database spends its time on lock churn instead of bookings. Latency explodes and even the winners time out.
- Ten shards: you get ten hot rows instead of one, plus rebalancing when some shards empty before others. The pool is only 500 seats.
- In front: a deliberately dumb atomic counter per pool (for example a Redis DECR). It hands out up to 500 "tickets to try" and rejects everyone else instantly, at well over 100,000 operations a second on one core.
- Behind it: only those admitted reach a single-writer allocator that assigns real berths in an ordinary transaction. The database only ever sees traffic proportional to inventory.
The Challenge:
A normal database row gets touched a few times a second. This train's Tatkal pool gets touched 280,000 times a second.
If we naively implement this as:
SELECT available_seats FROM pool WHERE train_id=X AND quota='TQ'
UPDATE pool SET available_seats = available_seats - 1
WHERE train_id=X AND quota='TQ' AND available_seats > 0...every single one of those 280,000 requests/sec fights for the SAME row lock. The database doesn't process bookings anymore — it processes lock queue churn. Latency explodes, timeouts cascade, and even the 500 legitimate winners can't get through the noise.
Splitting the row into shards (a classic hot-key fix) doesn't fully help here either: the total pool is only 500 seats. Split into 10 shards of 50 seats each, and you just create 10 hot rows instead of 1 — plus now you need cross-shard rebalancing when one shard empties before another.
The Solution:
%% title: Separate "can I even try?" from "give me the seat"
flowchart TD
REQ(["Booking request"]):::client
subgraph S1["Stage 1 · Fast admission check (Redis, in-memory, single-digit ms)"]
LUA@{ icon: "tech:redis", label: "Atomic Lua script on counter<br/>pool:12301:2026-08-10:3A:TQ<br/>(starts at 500)" }
CTR{"counter > 0?"}:::cache
LUA ==> CTR
end
REQ ==> LUA
CTR ==>|"yes · counter -= 1"| TTT(["TICKET_TO_TRY"]):::ok
CTR -->|"no"| SORRY(["SORRY_TRY_ANOTHER_TRAIN"]):::bad
subgraph S2["Stage 2 · Durable allocation (only the ~500 who passed Stage 1)"]
ALLOC["Single-writer, fully serialized allocator<br/>normal transactional UPDATE"]:::hot
DB@{ icon: "sd:database", label: "Real database<br/>assigns berth numbers" }
ALLOC ==> DB
end
TTT ==> ALLOC
class LUA cache
class DB store- Stage 1: a single atomic counter per pool. This single operation is O(1), lock-free from the caller's perspective, and Redis can execute ~100,000+ of these per second on a single core. It rejects 1,999,500 requests almost for free.
- Stage 2: only requests holding a "ticket to try" reach the real database. At this reduced volume (500, not 2,000,000), a single-writer, fully serialized allocator can safely assign real berth numbers with a normal transactional UPDATE — no contention left to fight.
WHY THIS WORKS:
- The hot key (the counter) is intentionally simple: one integer, one atomic decrement, no joins, no side effects
- 99.975% of load is absorbed by something that isn't a database at all
- The database only ever sees traffic proportional to actual inventory
About 280,000 requests a second running UPDATE … WHERE available_seats > 0 queue on one row lock until latency explodes. Instead an atomic Lua script on a pool counter set to 500 hands out 500 TICKET_TO_TRY results, rejects the other 1,999,500 in milliseconds with no durable storage, and only the admitted reach a single-writer allocator. A retry with the same attempt id is found in the admitted set and never burns a second seat.
# inventory/hot_pool_admission.py
"""
Fast admission control for a single hot (train, date, class, quota) pool.
Design principle: the pool counter must answer ONE question, as fast as
possible, for as many callers as possible. It does not allocate berths,
does not touch passenger data, and does not talk to a durable database.
"""
from dataclasses import dataclass
from typing import Optional
import time
ADMISSION_SCRIPT = """
-- KEYS[1]: pool counter key, e.g. "pool:12301:2026-08-10:3A:TQ"
-- ARGV[1]: this request's unique attempt id (for idempotent retries)
-- Idempotent retry: this attempt was already admitted, so do NOT
-- decrement again (a retried request must not burn a second seat)
if redis.call('SISMEMBER', KEYS[1] .. ':admitted', ARGV[1]) == 1 then
return -3
end
local remaining = tonumber(redis.call('GET', KEYS[1]))
if remaining == nil then
return -2 -- pool not initialized / booking not open yet
end
if remaining <= 0 then
return -1 -- sold out, reject immediately
end
redis.call('DECR', KEYS[1])
redis.call('SADD', KEYS[1] .. ':admitted', ARGV[1])
return remaining - 1
"""
@dataclass
class AdmissionResult:
admitted: bool
reason: str
remaining_hint: int = 0
class HotPoolAdmissionGate:
"""
Guards a single (train, date, class, quota) pool from the full
force of Tatkal traffic. This is the ONLY thing 2 million
requests are allowed to touch directly.
"""
def __init__(self, redis_client, metrics):
self.redis = redis_client
self.metrics = metrics
self._script_sha = None
async def initialize_pool(self, pool_key: str, seat_count: int):
"""Called once, moments before the Tatkal window opens."""
self._script_sha = await self.redis.script_load(ADMISSION_SCRIPT)
await self.redis.set(pool_key, seat_count, nx=True)
async def try_admit(self, pool_key: str, attempt_id: str) -> AdmissionResult:
"""
The ONLY call 2 million requests make. Must resolve in
single-digit milliseconds under load.
"""
result = await self.redis.evalsha(
self._script_sha, 1, pool_key, attempt_id
)
if result == -2:
self.metrics.increment("admission_not_open", pool_key)
return AdmissionResult(False, "BOOKING_NOT_OPEN")
if result == -1:
self.metrics.increment("admission_sold_out", pool_key)
return AdmissionResult(False, "SOLD_OUT")
if result == -3:
# Same attempt retried: it already holds its ticket to try
return AdmissionResult(True, "ALREADY_ADMITTED")
self.metrics.increment("admission_granted", pool_key)
return AdmissionResult(True, "PROCEED_TO_ALLOCATION", remaining_hint=result)Deep Dive 2: The Virtual Waiting Room — Taming the Thundering Herd (Week 3 & 4 Concepts: Backpressure & Flow Control, Thundering Herd)#
"A hot pool admission gate solves contention inside the system. But 2 million simultaneous connections can crush the edge before a single request even reaches that gate. We need a front door that only opens as fast as the house behind it can hold people."
The Challenge:
At 09:59:59, hundreds of thousands of browser tabs and app instances are all sitting on a "Search" or "Book Now" button, synchronized to the same clock. At 10:00:00.000, they all fire at once.
This is not "high average load." It is a near-instantaneous step function: from near-zero to 200,000+ req/sec in under a second. No amount of auto-scaling reacts fast enough — by the time new servers are up, the herd has already stampeded through the old ones and knocked them over.
Naive load balancing just spreads the stampede across more servers, all of which still try to hit the same 500-seat pool at the same instant.
The Solution:
%% title: A virtual waiting room with adaptive admission
sequenceDiagram
autonumber
participant C as Client
participant W as Waiting Room
participant B as Booking Core
C->>W: Connects before 10:00:00
Note over W: Assigned a fair, randomized queue position<br/>(NOT simply first-request-wins, which would reward<br/>the fastest scripts, not the fastest humans)
loop every 1s
B-)W: Reports live health: CPU, DB latency, payment gateway success rate
Note over W,B: Admission rate adapts every 1s<br/>"System healthy → admit 1,000/sec"<br/>"DB latency over 50 ms → 500/sec, over 150 ms → 150/sec"
end
C->>W: Polls or holds a WebSocket
W-->>C: Admitted in small controlled batches, proportional to what downstream can absorbKEY DESIGN DECISIONS:
- QUEUE POSITION IS RANDOMIZED WITHIN SHORT WINDOWS, NOT PURE FIFO — Pure "first byte wins" rewards whoever has the fastest script, not the fastest human — exactly the bot behavior we want to discourage. Bucket arrivals into small time windows (e.g. 250ms) and shuffle within each bucket.
- ADMISSION RATE IS ADAPTIVE, NOT FIXED — A fixed "admit 1,000/sec" is either too conservative (wasting capacity) or too aggressive (overloading the booking core the moment DB latency ticks up). Feed back real signals every second.
- ONE ACTIVE QUEUE SLOT PER SESSION — A user's session is bound to their device (Week 1 concepts: session store design). Opening 10 tabs or automating 10 requests from one account must not yield 10 queue positions.
- REJECTED USERS GET A HONEST, CHEAP ANSWER — "Sold out" or "still waiting, position 340,201" is returned from the edge in milliseconds — never a spinning loader that silently times out.
# waitingroom/queue_service.py
"""
Virtual Waiting Room: absorbs the thundering herd before it ever
reaches the booking core, and admits requests at a rate the
downstream system has actually signaled it can handle.
"""
from dataclasses import dataclass
from typing import Optional
from datetime import datetime, timedelta
import hashlib
@dataclass
class DownstreamHealth:
"""Live signal from the booking core, refreshed every second."""
db_latency_ms: float
payment_success_rate: float
booking_core_cpu_pct: float
@dataclass
class QueueTicket:
session_id: str
bucket: int # coarse time-bucket for fair shuffling
shuffle_key: str # deterministic but unpredictable ordering
issued_at: datetime
class AdaptiveWaitingRoom:
"""
Buffers arrivals and admits them to the booking core at a
rate derived from real downstream health, not a static guess.
"""
BASE_ADMIT_RATE = 1000 # requests/sec under healthy conditions
MIN_ADMIT_RATE = 100 # never fully stop; keep some flow moving
BUCKET_WIDTH_MS = 250
def __init__(self, queue_store, session_binder, metrics):
self.queue = queue_store # ordered structure (e.g. Redis sorted set)
self.sessions = session_binder # enforces one active slot per user
self.metrics = metrics
async def enqueue(self, session_id: str) -> QueueTicket:
"""
Called once per user session as they arrive. Idempotent:
calling twice for the same active session returns the
SAME ticket, never a second slot.
"""
existing = await self.sessions.get_active_ticket(session_id)
if existing:
return existing
now = datetime.utcnow()
bucket = int(now.timestamp() * 1000) // self.BUCKET_WIDTH_MS
# Shuffle key: hash of session + bucket, NOT arrival timestamp.
# This defeats "fastest script wins" while staying deterministic.
shuffle_key = hashlib.sha256(
f"{session_id}:{bucket}".encode()
).hexdigest()
ticket = QueueTicket(
session_id=session_id,
bucket=bucket,
shuffle_key=shuffle_key,
issued_at=now,
)
await self.queue.add(bucket, shuffle_key, session_id)
await self.sessions.bind_active_ticket(session_id, ticket)
self.metrics.increment("queue_enqueued")
return ticket
def compute_admit_rate(self, health: DownstreamHealth) -> int:
"""
Adaptive throttling: shrink the front door the moment the
booking core shows strain, rather than waiting for it to fail.
"""
rate = self.BASE_ADMIT_RATE
if health.db_latency_ms > 50:
rate *= 0.5
if health.db_latency_ms > 150:
rate *= 0.3
if health.payment_success_rate < 0.98:
rate *= 0.6
if health.booking_core_cpu_pct > 80:
rate *= 0.4
return max(self.MIN_ADMIT_RATE, int(rate))
async def run_admission_tick(self, health: DownstreamHealth):
"""
Runs every ~1 second. Pulls the next batch of tickets, in
fair shuffled order, and releases them into the booking core.
"""
admit_count = self.compute_admit_rate(health)
batch = await self.queue.pop_next(admit_count)
for session_id in batch:
await self.sessions.mark_admitted(session_id)
self.metrics.gauge("admit_rate", admit_count)
self.metrics.gauge("queue_depth", await self.queue.size())
return batchHot pool and waiting room
Q1/4 Why doesn't splitting the 500-seat pool row into 10 shards of 50 solve the contention?
Answer & explanation
Answer: B
The classic hot-key fix assumes a large keyspace to spread over. With only 500 seats, every shard is still hammered and uneven drain needs rebalancing.
Q2/4 Why is the check-and-decrement done inside one Lua script rather than a GET followed by a DECR from the app server?
Answer & explanation
Answer: D
Between a separate GET and DECR, thousands of callers can read the same value and all proceed, admitting more people than seats.
Q3/4 Why does the waiting room shuffle arrivals inside 250 ms buckets instead of serving pure first-come-first-served?
Answer & explanation
Answer: C
Within a bucket, a few milliseconds of script speed no longer buys a better position. Ordering across buckets still respects rough arrival time.
Q4/4 The booking core reports DB latency 200 ms and CPU 85%, with payment success healthy. Using
compute_admit_rate(base 1,000/s), what rate does the waiting room admit?Answer & explanation
Answer: D
1,000 x 0.5 (latency > 50) x 0.3 (latency > 150) x 0.4 (CPU > 80) = 60, which is raised to the MIN_ADMIT_RATE floor of 100. The floor keeps some flow moving instead of stalling the queue entirely.
Deep Dive 3: Seat Holds and the Booking Saga (Week 2 & 5 Concepts: Idempotency, Saga Pattern)#
"Once a request has a real chance at a seat, we hit the exact same trap UPI faces with money: there's a window between 'we've provisionally given you something' and 'you've actually paid for it,' and that window must be bulletproof."
Hold the seat, take the money
Priya holds berth 3A and is paying by UPI. How long should the hold last, what releases it, and what happens if her bank confirms the payment 30 seconds after the hold's timer ran out?
One strong answer
- A hold with a timer: the berth is invisible to the pool while held, and the hold expires after a few minutes (the lesson uses 5) so abandoned bookings can't starve real demand.
- A saga: hold → pay → issue the PNR. If payment fails or times out, compensation returns the seat to the pool and refunds any money captured.
- One key per attempt: an idempotency key covers the hold, the payment and the PNR, so a double tap on Pay charges once.
- The late confirmation: the timer only releases holds still waiting for payment. Once Priya pays, the attempt is PAYMENT_INITIATED and the berth waits for the real outcome: success issues the PNR, a timeout goes to reconciliation, failure releases the seat and refunds. No phantom PNR, no resold seat under her payment, and no silently lost money.
The Challenge:
Between "you've reached the front of the queue and picked seat 24, berth 3A" and "your payment cleared," several seconds to a few minutes pass — Priya fills passenger details, enters an Aadhaar OTP, chooses UPI, and confirms in her banking app.
During that window
- The seat CANNOT be given to anyone else (or we oversell)
- The seat CANNOT be held forever (abandoned carts would starve real demand — remember, 1,999,500 others want this seat)
- If payment fails or times out, the hold must release automatically
- If Priya double-clicks "Pay Now," she must be charged once, not twice
- If the payment gateway is slow to confirm, we must not guess wrong in either direction — no phantom PNR, no silently lost money
The Solution:
%% title: Booking state machine (a saga with a TTL) stateDiagram-v2 [*] --> ADMITTED ADMITTED --> SEAT_HELD: Selects seat SEAT_HELD --> HOLD_EXPIRED: TTL expires SEAT_HELD --> PAYMENT_INITIATED: Details + OTP SEAT_HELD --> HOLD_RELEASED: Explicitly abandons PAYMENT_INITIATED --> PNR_CONFIRMED: Success PAYMENT_INITIATED --> COMPENSATING_RELEASE: Failure PAYMENT_INITIATED --> DEEMED_PENDING: Timeout ADMITTED: ADMITTED · passed the waiting room SEAT_HELD: SEAT_HELD · TTL 5 minutes HOLD_EXPIRED: HOLD_EXPIRED · seat returned to pool HOLD_RELEASED: HOLD_RELEASED · seat returned to pool PAYMENT_INITIATED: PAYMENT_INITIATED PNR_CONFIRMED: PNR_CONFIRMED COMPENSATING_RELEASE: COMPENSATING RELEASE DEEMED_PENDING: DEEMED_PENDING · reconcile with gateway, then confirm OR refund+release class ADMITTED,SEAT_HELD,PAYMENT_INITIATED core class PNR_CONFIRMED ok class HOLD_EXPIRED,HOLD_RELEASED,COMPENSATING_RELEASE bad class DEEMED_PENDING queue
- SEAT_HELD: the held berth is INVISIBLE to the pool counter — it's already gone.
- COMPENSATION (the Saga's rollback): if payment fails after the seat was held, the seat is atomically returned to the pool counter AND, if money was captured, a refund is triggered — automatically, no ticket.
# booking/saga_orchestrator.py
"""
Booking Saga Orchestrator.
Coordinates seat hold -> passenger/OTP -> payment -> PNR as one
logical transaction spanning multiple services, with compensating
actions if any step fails. Modeled the same way UPI's debit/credit
saga is: hold first, confirm second, and ALWAYS have a clean way back.
"""
from dataclasses import dataclass
from typing import Optional
from enum import Enum
from datetime import datetime, timedelta
import uuid
class BookingState(Enum):
ADMITTED = "admitted"
SEAT_HELD = "seat_held"
PAYMENT_INITIATED = "payment_initiated"
PNR_CONFIRMED = "pnr_confirmed"
HOLD_EXPIRED = "hold_expired"
HOLD_RELEASED = "hold_released"
DEEMED_PENDING = "deemed_pending"
COMPENSATED = "compensated"
@dataclass
class BookingAttempt:
attempt_id: str
pool_key: str # train:date:class:quota
state: BookingState
seat_ref: Optional[str] = None
payment_ref: Optional[str] = None
pnr: Optional[str] = None
hold_expires_at: Optional[datetime] = None
idempotency_key: str = ""
class BookingSagaOrchestrator:
"""
Orchestrates the multi-step booking saga for one attempt.
Guarantees:
1. IDEMPOTENCY: retrying with the same idempotency_key never
double-charges or double-allocates a seat.
2. BOUNDED HOLDS: a held seat is always released within TTL,
either by success, explicit abandonment, or expiry. (The TTL
job only releases attempts still in SEAT_HELD; a DEEMED_PENDING
hold is resolved by reconciliation instead, so a payment that
turns out to have succeeded never lands on a resold seat.)
3. COMPENSATION: any failure after a hold releases the seat and
refunds any captured payment — automatically.
"""
HOLD_TTL = timedelta(minutes=5)
def __init__(self, db, seat_pool, payment_gateway, pnr_generator,
notifier, expiry_scheduler):
self.db = db
self.pool = seat_pool
self.payments = payment_gateway
self.pnr_generator = pnr_generator
self.notifier = notifier
self.expiry_scheduler = expiry_scheduler
async def hold_seat(self, pool_key: str, idempotency_key: str) -> BookingAttempt:
"""Step 1: convert an admission ticket into a real, TTL-bound hold."""
existing = await self.db.get_by_idempotency_key(idempotency_key)
if existing:
return existing # same request retried; return the same attempt
seat_ref = await self.pool.allocate_from_pool(pool_key)
if seat_ref is None:
return BookingAttempt(
attempt_id=str(uuid.uuid4()),
pool_key=pool_key,
state=BookingState.HOLD_EXPIRED,
idempotency_key=idempotency_key,
)
attempt = BookingAttempt(
attempt_id=str(uuid.uuid4()),
pool_key=pool_key,
state=BookingState.SEAT_HELD,
seat_ref=seat_ref,
hold_expires_at=datetime.utcnow() + self.HOLD_TTL,
idempotency_key=idempotency_key,
)
await self.db.save(attempt)
# Schedule an automatic release exactly at TTL — never rely on
# a client callback to free an abandoned seat.
await self.expiry_scheduler.schedule(
attempt.attempt_id, run_at=attempt.hold_expires_at
)
return attempt
async def confirm_with_payment(
self, attempt_id: str, payment_token: str
) -> BookingAttempt:
"""Step 2: attempt payment for a held seat, then confirm or compensate."""
attempt = await self.db.get(attempt_id)
if attempt.state != BookingState.SEAT_HELD:
return attempt # already resolved one way or another
if datetime.utcnow() > attempt.hold_expires_at:
await self._release_hold(attempt, reason="EXPIRED_BEFORE_PAYMENT")
return attempt
attempt.state = BookingState.PAYMENT_INITIATED
await self.db.save(attempt)
result = await self.payments.charge(
token=payment_token,
reference=attempt.attempt_id,
idempotency_key=attempt.idempotency_key,
)
if result.status == "SUCCESS":
pnr = await self.pnr_generator.generate(attempt)
attempt.pnr = pnr
attempt.payment_ref = result.reference
attempt.state = BookingState.PNR_CONFIRMED
await self.db.save(attempt)
await self.notifier.send_confirmation(attempt)
return attempt
if result.status == "TIMEOUT":
# Uncertain outcome — do NOT guess. Park it for reconciliation,
# exactly like UPI's "deemed success" state.
attempt.state = BookingState.DEEMED_PENDING
await self.db.save(attempt)
await self.expiry_scheduler.schedule_reconciliation(
attempt.attempt_id, check_in=timedelta(minutes=10)
)
return attempt
# Clean failure — compensate immediately
await self._release_hold(attempt, reason="PAYMENT_FAILED")
return attempt
async def _release_hold(self, attempt: BookingAttempt, reason: str):
"""The Saga's compensating transaction: give the seat back."""
await self.pool.release_to_pool(attempt.pool_key, attempt.seat_ref)
attempt.state = BookingState.HOLD_RELEASED
await self.db.save(attempt)
if attempt.payment_ref:
await self.payments.refund(attempt.payment_ref)
async def reconcile_deemed_pending(self, attempt_id: str):
"""
Called by the background reconciliation worker for any
booking stuck in DEEMED_PENDING — resolves it one way,
permanently, based on the payment gateway's true record.
"""
attempt = await self.db.get(attempt_id)
actual = await self.payments.check_status(attempt.attempt_id)
if actual.status == "SUCCESS":
pnr = await self.pnr_generator.generate(attempt)
attempt.pnr = pnr
attempt.payment_ref = actual.reference
attempt.state = BookingState.PNR_CONFIRMED
await self.db.save(attempt)
await self.notifier.send_confirmation(attempt)
elif actual.status == "PENDING":
# Gateway still doesn't know: check again later, never guess
await self.expiry_scheduler.schedule_reconciliation(
attempt.attempt_id, check_in=timedelta(minutes=10)
)
else:
await self._release_hold(attempt, reason="RECONCILED_AS_FAILED")Deep Dive 4: Aadhaar, OTP, and the War on Bots (Week 9 Concepts: Security Architecture)#
"Every seat a bot or a bulk-booking agent grabs in the first ten seconds is a seat a genuine passenger never sees. Since mid-2025, Indian Railways has treated this as a core architecture problem, not just a policy problem."
The Challenge:
- 3.03 crore (30.3 million) suspicious user IDs deactivated
- Crores more IDs flagged for re-verification
- Thousands of suspicious email domains blocked
- 4.18 lakh (418,000) suspicious PNRs linked to cyber-crime complaints
- 501 complaints filed on the National Cyber Crime Portal
The threat isn't "hackers breaking encryption." It's:
- Scripts that auto-fill and auto-submit faster than any human
- Agents/touts who book Tatkal tickets in bulk to resell at a premium
- Farms of throwaway accounts + disposable emails to dodge per-user booking limits
- Data-center IPs (not real home/mobile networks) hammering the booking API directly, bypassing the normal app/website entirely
The Solution:
- Layer 1: Identity verification (regulatory, since mid-2025)
- Aadhaar authentication required for Tatkal on web/app (effective 1 July 2025)
- Aadhaar-based OTP mandatory for Tatkal — web, app, AND counter/agent bookings (effective 15 July 2025)
- Raises the cost of running throwaway accounts sharply
- Layer 2: Time-window restrictions
- Authorized agents blocked from booking in the first 30 minutes of the Tatkal window (AC: 10:00-10:30, non-AC: 11:00-11:30)
- This protects the highest-value first seconds for individual travelers, while still letting agents serve customers afterward
- Layer 3: Session & device binding
- One active login per user, and a cap on Tatkal bookings per user per window (our design choice; IRCTC's published rule is at most 4 passengers per PNR)
- Device fingerprint bound to the session; re-login invalidates old sessions rather than allowing parallel sessions
- Layer 4: Behavioral & velocity signals
- Fill-speed analysis: real humans take measurable time to type passenger names and read OTP screens; scripts don't
- Request velocity per account, per device, and per IP/ASN
- Data-center / hosting-provider IP ranges scored higher risk than residential or mobile carrier IPs
- Repeated failed-then-retried patterns across many "different" accounts sharing a device fingerprint or payment instrument
# security/tatkal_risk_gate.py
"""
Risk gate specific to the Tatkal booking window. Runs BEFORE the
booking saga is allowed to hold a seat, so that flagged traffic
never even competes for real inventory.
"""
from dataclasses import dataclass
from typing import List, Tuple
from datetime import datetime, time
@dataclass
class TatkalRiskSignals:
user_id: str
device_fingerprint: str
is_authorized_agent: bool
aadhaar_verified: bool
otp_verified: bool
# Velocity
requests_per_min_this_device: int
accounts_seen_on_this_device_today: int
# Network
is_datacenter_ip: bool
ip_request_rate_per_sec: float
# Behavioral
form_fill_time_ms: int # time between page load and submit
tatkal_bookings_today: int # this user, this opening window
class TatkalRiskGate:
"""
Fast, explainable rules run first (cheap, deterministic);
an ML model catches subtler patterns the rules miss.
Target decision latency: < 50ms, same discipline as a real-time
payment fraud check.
"""
MIN_HUMAN_FILL_TIME_MS = 1200 # a script can submit in <100ms
def __init__(self, ml_model, agent_window_config):
self.model = ml_model
self.agent_window = agent_window_config # (start, end) per class
async def evaluate(
self, signals: TatkalRiskSignals, now: datetime, travel_class: str
) -> Tuple[str, List[str]]:
"""
Returns: (decision, triggered_rules)
decision: "ALLOW", "BLOCK", "STEP_UP" (extra OTP / CAPTCHA)
"""
triggered: List[str] = []
# RULE 1: Identity verification is mandatory, not optional
if not (signals.aadhaar_verified and signals.otp_verified):
return "BLOCK", ["IDENTITY_NOT_VERIFIED"]
# RULE 2: Agent time-window restriction
if signals.is_authorized_agent:
start, end = self.agent_window.get(travel_class)
if start <= now.time() <= end:
return "BLOCK", ["AGENT_WINDOW_RESTRICTED"]
# RULE 3: One Tatkal booking per user per window
if signals.tatkal_bookings_today > 0:
return "BLOCK", ["ALREADY_BOOKED_THIS_WINDOW"]
# RULE 4: Device fan-out — many accounts, one device, same morning
if signals.accounts_seen_on_this_device_today > 3:
triggered.append("DEVICE_ACCOUNT_FANOUT")
# RULE 5: Network origin
if signals.is_datacenter_ip:
triggered.append("DATACENTER_IP")
if signals.ip_request_rate_per_sec > 5:
triggered.append("HIGH_IP_VELOCITY")
# RULE 6: Inhuman speed
if signals.form_fill_time_ms < self.MIN_HUMAN_FILL_TIME_MS:
triggered.append("SUBHUMAN_FILL_SPEED")
# ML model catches the combinations rules can't enumerate
ml_score = await self.model.predict(signals)
if ml_score > 0.9 or len(triggered) >= 3:
return "BLOCK", triggered + ["ML_HIGH_RISK"]
if ml_score > 0.6 or len(triggered) >= 1:
return "STEP_UP", triggered
return "ALLOW", triggeredBooking saga and anti-bot defenses
Q1/4 The payment gateway call times out. Why does the saga move to DEEMED_PENDING instead of releasing the seat right away?
Answer & explanation
Answer: A
A timeout is not a failure. The reconciliation worker checks the gateway's true record, then either confirms the PNR or refunds and releases.
Q2/4 Why is seat release scheduled server-side at the hold TTL rather than triggered by the app when the user leaves?
Answer & explanation
Answer: C
Bounded holds are a guarantee only if the server enforces them. Relying on a client callback would let abandoned carts starve real buyers.
Q3/4 Priya taps "Pay Now" twice within a second. What prevents a double charge?
Answer & explanation
Answer: B
The client generates one idempotency key per attempt and every downstream call (hold, payment, PNR) is keyed on it: no second charge, no second seat.
Q4/4 A Tatkal request has verified Aadhaar OTP, comes from a data-center IP, filled the form in 300 ms, and the ML score is 0.4. What does
TatkalRiskGatedecide?Answer & explanation
Answer: D
Hard rules pass (verified identity, not an agent, no prior booking). Two triggered soft rules are below the block threshold of 3, and the ML score is under 0.9, so the user gets extra verification.
Phase 5: Scaling and Edge Cases (5 minutes)#
"Walk me through your Tatkal-day playbook — what actually happens in the hours around 10 AM?"
Scaling Strategies#
- T-30 minutes (09:30 AM)
- Pre-scale booking-core compute for known hot trains (historical demand data predicts which routes will spike)
- Warm caches for search/availability on those routes
- Pre-authenticate sessions: users who logged in early get their Aadhaar OTP validated ahead of time where policy allows, so the critical path at 10:00:00 is shorter
- T-5 minutes (09:55 AM)
- Freeze non-critical deployments and config changes
- Initialize hot-pool counters (Deep Dive 1) for every train opening Tatkal quota this run
- Waiting room admission starts accepting connections, holding everyone at the door
- T-0 (10:00:00 AM)
- Waiting room begins admitting at the adaptive rate (Deep Dive 2)
- Search/enquiry traffic is served entirely from the read plane — it never touches booking-core capacity
- Booking core only ever sees traffic proportional to real inventory, not raw demand
- T+15 minutes (10:15 AM)
- Hot pools for popular trains are exhausted; counters read zero (the hottest trains sell out within about 5 minutes, once the last holds are paid or lapse)
- Waiting room shifts remaining queued users straight to "sold out"
- Auto-scale booking-core compute back down; read plane keeps absorbing the long tail of enquiries
- Ongoing, all day
- Waitlist promotion runs as cancellations arrive, walking the
waitlist_queuein strict order (WL → RAC → CNF) - First chart preparation (by 8 PM the previous day, or ~10 hours before departure) finalizes the passenger list and runs one last promotion pass
- Waitlist promotion runs as cancellations arrive, walking the
Edge Cases#
Edge case 1: Payment succeeds after the hold already expired
Problem: Priya's payment gateway call is slow. Her 5-minute hold timer runs out, and her bank confirms the payment 30 seconds later.
Solution:
- The TTL job only releases attempts still in SEAT_HELD (Deep Dive 3). Priya's moved to PAYMENT_INITIATED when she paid, so her berth is never resold under an in-flight payment
- The late success simply confirms her PNR; a gateway timeout parks the attempt in DEEMED_PENDING, and reconciliation confirms or releases and refunds
- A payment that arrives with no live attempt at all (say, a UPI collect approved after the attempt was released) is reconciled and auto-refunded with priority, plus a clear notification: never a silent loss
Edge case 2: Double-submit from an impatient user
Problem: The confirm button is tapped twice within a second because the UI felt slow.
Solution:
- Client generates one
idempotency_keyper booking attempt - Every downstream call (hold, payment, PNR) is keyed on it
- Second submission returns the SAME result as the first — no second charge, no second seat consumed
Edge case 3: Last-minute cancellation triggers a waitlist promotion race
Problem: A confirmed passenger cancels the day before departure (before the chart is prepared). Multiple waitlisted passengers could be considered for that one seat.
Solution:
- Promotion is a single-writer, ordered walk over
waitlist_queue— never a parallel "first request wins" race - Promoted passengers are notified via SMS before the seat is irreversibly reassigned to someone further down the list
Edge case 4: The write plane and read plane accidentally share load
Problem (real, documented pattern): if search/enquiry traffic and booking traffic share infrastructure, a spike in one degrades the other. IRCTC's website has visibly gone down or been throttled during Tatkal hours on multiple occasions (e.g. December 2024, users saw "e-ticketing service will not be available for the next one hour" precisely during the Tatkal window).
Lesson learned:
- Enquiry and booking MUST be physically separate service tiers, not just logically separate endpoints on shared infrastructure
- Rate limit the enquiry path aggressively — it's read-only and idempotent, so aggressive throttling costs nothing but a stale number on screen
- Give the booking path its own dedicated, pre-scaled capacity that enquiry traffic can never consume
Edge case 5: A single hot train's chaos leaking into other trains
Problem: If seat inventory for all trains lives in one shared database cluster, one popular train's 280,000 req/sec burst can starve query capacity for the other 12,999 trains running that day.
Solution:
- Hot pools are isolated per (train, date) — a busy Rajdhani's admission gate runs on infrastructure that a quiet local passenger train's booking never touches
- The Seat Inventory Service intentionally does NOT share a connection pool or lock table across unrelated trains
Scaling and edge cases
Q1/3 A confirmed passenger cancels the day before departure and several waitlisted passengers could take the seat. How is the race avoided?
Answer & explanation
Answer: D
Promotion is an ordered, serialized walk, never a parallel "first request wins" race, so the right person gets the seat exactly once.
Q2/3 A popular Rajdhani's 280,000 req/s burst is starving bookings for quiet passenger trains on the same database. What is the fix?
Answer & explanation
Answer: B
The requirement is that one hot train never degrades the other 12,999. Isolation, not more shared capacity, is what makes that true.
Q3/3 Priya's payment is in flight when her 5-minute hold timer runs out, and her bank confirms the payment 30 seconds later. What should happen?
Answer & explanation
Answer: C
The TTL job only releases attempts in SEAT_HELD; hers is in PAYMENT_INITIATED, so the berth waits for the payment's real outcome. Every captured rupee must end in a PNR or a refund.
Phase 6: Monitoring and Operations#
"For a system where 'temporarily down' during Tatkal means real families can't travel, observability during the opening minutes matters more than almost anything else we've built."
Monitoring Dashboard#
Admission funnel (last 60 seconds, this train)
| Stage | Count |
|---|---|
| Requests received | 1,998,412 |
| Admitted to booking core | 14,207 |
| Seats successfully held | 512 |
| PNRs confirmed | 487 |
Write plane health
- DB latency (p99): 38 ms
- Payment success: 98.7%
- Admit rate (current): 1,000/sec (full rate: DB < 50 ms, payments ≥ 98%)
Read plane health
- Cache hit rate: 99.4%
- Enquiry latency p99: 60 ms
- Enquiry rate: 6,700/sec
Fraud gate (this train, this window)
- Blocked — identity not verified: 41,203
- Blocked — agent window restriction: 2,891
- Blocked — device/IP velocity rules: 18,447
- Step-up (extra verification) triggered: 6,120
Alerts (last 15 minutes)
- 10:00:00 (green) — Waiting room opened, admitting at adaptive rate
- 10:02:14 (amber) — DB latency crossed 50ms, admit rate halved to 500/sec
- 10:00:10 (green) — Every seat in train 12301's hot pool is held
- 10:04:50 (green) — Last holds paid or lapsed and re-sold: train 12301 sold out
- 10:15:00 (green) — Booking core auto-scaled down, read plane unaffected
SLOs for the Tatkal System#
SLO 1: Zero oversell
- Target: 100% — the pool counter and the durable allocator never disagree on how many seats have been given out
- Error budget: 0. This is non-negotiable, exactly like UPI's money safety guarantee.
SLO 2: Admission decision latency
- Target: p99 < 200ms for "you're in the queue" or "sold out"
- Current: p99 = 140ms
SLO 3: Read/write plane isolation
- Target: enquiry traffic causes zero measurable increase in booking path latency, even at lakhs of enquiries a minute
SLO 4: Payment-to-seat integrity
- Target: 100% of captured payments resolve to either a confirmed PNR or a completed refund within 30 minutes
Reading the Tatkal dashboard
Q1/3 The funnel shows 512 seats held but only 487 PNRs confirmed. What happened to the difference?
Answer & explanation
Answer: B
Held can exceed the quota because seats from failed or expired holds are returned by the saga's compensation and held again by someone else. The zero-oversell SLO is about confirmed allocations, not hold attempts.
Q2/3 DB latency crosses 50 ms and the admit rate is halved automatically. Why throttle admission instead of letting requests queue at the database?
Answer & explanation
Answer: B
Queueing at the database adds latency and timeouts for everyone already admitted, including people mid-payment. The waiting room is the cheap place to wait, and adaptive admission keeps the expensive core fast.
Q3/3 SLO 4 promises every captured payment ends as a PNR or a refund within 30 minutes. Which mechanism guarantees the refund side?
Answer & explanation
Answer: C
Compensation handles failures the system sees. Reconciliation catches the ones it doesn't, such as a late confirmation after a hold expired, so every rupee is accounted for.
Final Challenge: Fifty Hot Trains at 10:00:00#
The interviewer taps the whiteboard. "Now make it real."
"You've designed for one hot train. But Tatkal for AC classes opens at 10:00 for every train at once, and on a festival weekend fifty of them are as hot as the one we discussed. How does your design change?"
- Fifty pools open in the same second, each with hundreds of seats and huge demand
- Most users are after one specific train, but switch quickly when it sells out
- A failure in one train's booking must not spill into the others
- Total capacity is finite and has to be shared sensibly
Final challenge · Quest check
Q1/4 Fifty hot trains open at once. How should their seat counters be placed?
Answer & explanation
Answer: B
Pools are independent, so their counters should be too. Spreading them, and isolating the hottest, keeps any single core from carrying several 280,000-requests-a-second bursts.
Q2/4 Why is a single-writer allocator per pool not a throughput problem?
Answer & explanation
Answer: B
The counter absorbs 99.975% of requests. The allocator handles a few hundred admitted attempts per pool, so serialising them costs nothing and buys certainty: no seat is ever allocated twice.
Q3/4 Train A sells out in about five minutes and its rejected users rush to search other trains. Which part of the system absorbs that?
Answer & explanation
Answer: C
Searching is read-only and can be a moment stale, so caches and rate limits handle the spike. Keeping it off the write plane is exactly why the two are physically separate.
Q4/4 1.4 million requests arriving in five seconds is about how many per second?
Answer & explanation
Answer: C
1,400,000 ÷ 5 = 280,000 a second for one train, against about 620 bookings a second for the whole national platform on its record minute. Without admission control, one train's burst would dwarf everything else.
Interview Conclusion#
"Solid. A few rapid-fire questions."
"Why not just add more database servers? Isn't this a capacity problem?"
"Only 500 seats exist. No amount of extra capacity changes that fact — it only changes how fast you can tell 1,999,500 people 'no.' The real engineering problem is admission control: reject the impossible cases as early and cheaply as possible, and only let real contention reach real inventory."
"Why is search/availability a completely separate system from booking?"
"Because their failure modes are opposite. Search can be stale by a second and nobody's hurt — a cache serving slightly outdated availability is a fine trade-off. Booking cannot be stale by even a millisecond, or two people get sold the same berth. Mixing them means the system that can tolerate staleness ends up dictating the availability of the system that cannot."
"What's the single most important design decision here?"
"Making the hot pool counter dumb on purpose. It does one atomic operation and nothing else — no joins, no passenger data, no payment logic. Every extra responsibility you bolt onto a hot key is extra work multiplied by 280,000 requests per second. The durable, feature-rich logic only runs for the ~500 requests that actually earned the right to reach it."
"If you were improving today's real IRCTC system, what would you change?"
"Based on documented outages during Tatkal hours: stronger physical separation between the enquiry and booking tiers, so a search traffic spike can never again produce a 'service unavailable for the next hour' message during the exact minute people need booking most. I'd also push the identity-verification and OTP steps to happen before the queue, not during it, so verified users don't burn their queue position on an avoidable extra round-trip."
Summary: Concepts Applied from 10-Week Course#
- Week 1: Data at scale
- Hot keys & skew — the 500-seat pool as the system's hottest row
- Rate limiting — per-user, per-device, per-IP admission limits
- Session store design — one active session/queue slot per user
- Week 2: Failure-first design
- Idempotency —
idempotency_keyacross hold, payment, and PNR - Timeouts — bounded seat holds (5-minute TTL), bounded payment waits before falling back to reconciliation
- The flash-sale pattern — claim inventory, then checkout, mirrors exactly what Week 2's capstone (flash sale for a retailer) covers
- Week 3: Messaging & async processing
- Backpressure & flow control — the waiting room's adaptive admit rate, driven by live downstream signals
- Queue vs stream — ordered
waitlist_queuefor fair promotion
- Week 4: Caching
- Thundering herd mitigation — the entire waiting-room design exists to prevent one
- Cache invalidation — availability snapshots refreshed on a short TTL, decoupled from the authoritative pool counter
- Week 5: Consistency & coordination
- Saga pattern — hold → pay → confirm, with compensation on failure
- Single-writer serialization — durable allocation for the reduced, post-admission-gate volume
- Leader-election-style ownership — one authoritative writer per pool
- Week 9: Multi-tenancy & security
- Security architecture — layered identity, time-window, and behavioral defenses against bots and bulk-booking agents
- Real regulatory response — Aadhaar OTP mandate, agent restrictions
- Week 10: Production readiness
- SLOs — zero-oversell, admission latency, plane isolation targets
- Capacity planning — the Tatkal-day playbook, pre-scale/post-scale
- Incident learnings — documented real-world outages as case studies
Why IRCTC's Tatkal System Matters#
Scale
- Up to 2 million attempts against 500 seats on a single hot train
- 37,410 tickets/minute nationally at record peak (16 Aug 2025)
A different shape of hard
- UPI's hard problem is moving money correctly between many parties
- Netflix's hard problem is delivering bytes efficiently at scale
- IRCTC's Tatkal problem is admission control: making 99.975% of all traffic disappear cheaply, fairly, and in milliseconds, so the remaining 0.025% can be served perfectly
Human stakes
- A failed booking can mean a missed job interview, a missed funeral, a missed wedding — Tatkal exists precisely for urgent, last-minute travel
- A double-booked seat means two real families in conflict at a real train door
Ongoing evolution
- Aadhaar-OTP authentication (2025) reshaped identity verification
- Agent booking windows (2025) reshaped fairness policy
- The revamped site (live 15 July 2026) lifted capacity to 1.5 lakh+ bookings/minute, up from ~32,000
"The hardest part of this system was never the database. It was accepting that almost every request must fail, and designing for that failure to be instant, honest, and fair."
Self-Assessment Checklist#
After studying this case study, you should be able to:
Architecture:
- Explain why search/enquiry and booking must be architecturally separate planes
- Design a hot-key admission gate that protects a durable database from extreme contention
- Trace the full booking flow from queue admission to PNR confirmation
Distributed Systems:
- Implement an adaptive virtual waiting room driven by live downstream health signals
- Design a Saga-based hold-then-pay flow with automatic compensation
- Explain why single-writer serialization can outperform naive horizontal sharding when the total inventory is small
Scale:
- Calculate contention ratios and peak instantaneous request rates for flash-demand events
- Explain why storage is rarely the bottleneck in extreme-contention systems
- Design a Tatkal-day capacity playbook (pre-scale, admit, exhaust, de-scale)
Security:
- Design layered anti-bot defenses: identity verification, time-windows, velocity, behavior
- Explain the real-world rationale behind Aadhaar OTP and agent-window restrictions
Operations:
- Define SLOs appropriate for a zero-oversell, admission-control-dominated system
- Monitor an admission funnel end-to-end, from raw requests to confirmed PNRs
- Learn from real documented outages where read and write traffic shared infrastructure
Sources#
Statistics and Scale:
- Read
The Abstract Engineer (Medium) — How IRCTC Handles Crores of Users During Tatkal Booking: https://medium.com/thesystemdesign/how-irctc-handles-crores-of-users-during-tatkal-booking-f5209531c5cd
- Read
IANS — IRCTC handles 18 lakh daily tickets (daily average and record figures): https://ianslive.in/irctc-delivers-58-crore-meals-a-year-handles-18-lakh-daily-tickets--20260603170101
- Read
Swarajya — Tatkal bookings faster after revamped IRCTC website (capacity figures): https://swarajyamag.com/infrastructure/tatkal-ticket-bookings-get-faster-after-railways-launched-revamped-irctc-beta-website-on-15-july
- Read
The Hitavada — Reservation charts to be ready one day in advance: https://www.thehitavada.com/Encyc/2025/12/27/now-railway-reservation-charts-to-be-ready-one-day-in-advance.html
- Read
New Kerala — IRCTC Blocks 3 Crore Suspicious IDs in 2025: https://www.newkerala.com/news/a/irctc-sets-new-records-online-ticket-booking-blocks-942.htm
Architecture and Technical Design:
- Read
GetSDEReady — IRCTC System Design: Scalable Train Ticket Booking Architecture: https://getsdeready.com/irctc-system-design/
- Read
System Design Academy — IRCTC System Design Interview: Tatkal and Train Booking at Scale: https://www.systemdesign.academy/interview/design-irctc
- Read
Shubham Shrivas (Medium) — IRCTC Tatkal Ticket Booking, High-Level System Design: https://shubhamshrii.medium.com/irctc-tatkal-ticket-booking-high-level-system-design-9a78eb4b0afd
- Read
Tushar Khandelwal (Medium) — IRCTC System Design: https://medium.com/@khandelwaltushar2002/irctc-system-design-398d34b0501c
- Blog
Naveen Kamaraj (Medium) — Uncover the Engineering Areas Behind IRCTC Ticket Booking: https://naveenkamaraj.medium.com/uncover-the-engineering-areas-behind-irctc-ticket-booking-d7872cde3520
- Blog
CIO.com — How IRCTC's New Servers Make Bookings and Enquiries Easier: https://www.cio.com/article/218286/how-irctc-s-new-servers-make-bookings-and-enquiries-easier.html
- Read
TechyUltra — How Tatkal Ticket Booking System Works: Technology Behind IRCTC System: https://techyultra.com/tatkal-ticket-booking-system/
Policy and Regulatory Changes (2025):
- Book
Press Information Bureau, Government of India — Only Aadhaar-Authenticated Users Can Book Tatkal Tickets on IRCTC Website and App from July 1: https://www.pib.gov.in/PressReleasePage.aspx?PRID=2135694®=48&lang=2
- Read
Zee News — IRCTC Tatkal Ticket Booking Rules 2025: Aadhaar OTP Mandatory From July 1, No Agent Bookings in First 30 Minutes For AC Tickets: https://zeenews.india.com/mobility/irctc-tatkal-ticket-booking-rules-2025-aadhaar-otp-mandatory-from-july-1-no-agent-bookings-in-first-30-minutes-for-ac-tickets-2917337.html
Real Incidents (Outages During Tatkal Hours):
- Blog
Upstox News — IRCTC Website Crashes Again During Peak Tatkal Hours: https://upstox.com/news/business-news/latest-updates/irctc-website-crashes-again-during-peak-tatkal-hours-users-slam-ticketing-downtime/article-136998/
- Read
The Cyber Express — IRCTC Outage Disrupts Tatkal Ticket Bookings Across India: https://thecyberexpress.com/irctc-outage-disrupts-tatkal-bookings/
Further Reading#
Official Sources:
- Read
IRCTC Official Website: https://www.irctc.co.in/
- Read
Indian Railways / Ministry of Railways Press Releases: https://www.pib.gov.in/ (search "Tatkal")
- Read
CRIS (Centre for Railway Information Systems): the organization that builds and operates PRS and IRCTC's backend ticketing platforms
Related Systems to Study:
- Read
Ticketmaster / BookMyShow — concert and event ticketing at flash-sale scale: nearly identical admission-control and hot-inventory problems
- Boss
Amazon Prime Day / e-commerce flash sales: this course's own Week 2 capstone ("Flash Sale for MegaMart") covers the same claim-then-checkout pattern
(opens in new tab) - Read
Airline GDS systems (Amadeus, Sabre): seat inventory allocation at a different but related scale
- Side quest
UPI (Bonus Problem 1 in this series): the closest sibling problem — atomic, reversible, hold-then-confirm transactions under extreme concurrency
(opens in new tab)
Foundational Concepts (this course):
- Lesson
Week 1, Day 4 — Hot Keys and Skew
(opens in new tab) - Boss
Week 2, Capstone — Flash Sale for MegaMart
(opens in new tab) - Lesson
Week 3, Day 3 — Backpressure and Flow Control
(opens in new tab) - Lesson
Week 4, Day 3 — Thundering Herd
(opens in new tab) - Lesson
Week 5, Day 2 — Distributed Transactions: The Saga Pattern
(opens in new tab)
"Two million people, five hundred seats, ten seconds. The system that handles this gracefully isn't the one with the biggest database — it's the one that knows, instantly and honestly, who to turn away."
this lesson in shorts · 3
play them all →One idea from this lesson in about a minute each, made for your phone.
- 0/7stages cleared
Follow-up drills
0 of 10 practisedYou presented IRCTC Tatkal Ticket Booking System. Now the interviewer pushes back. Answer each follow-up out loud or in the box, then reveal a strong answer.
“Two million attempts for 500 seats. Is this a storage problem?”
Pushes on · finding the real bottleneck in the numbers
Go deeper · reading trail
all reading →Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.
Building Waiting Room on Workers and Durable Objects
Fabienne Semeria, George Thomas, Mathew Jacob · 2021- What
- How Cloudflare built its Waiting Room product, first offered for COVID-19 vaccination sites.
- Key idea
- When users exceed a site's limit, new arrivals get a cookie and a queue page with an estimated wait; Workers and Durable Objects keep global counts to admit them in order.
- Why here
- The virtual waiting room deep dive of this quest: hold the herd at the edge and admit at the rate the booking system can take.
Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers (Part I)
Emil Stolarsky · 2017- What
- How Shopify kept recurring flash sales from taking down shared database shards.
- Key idea
- A checkout throttle in Lua on the Nginx edge applies back pressure, letting buyers into checkout only as fast as it can process them.
- Why here
- Tatkal at 10:00 is a flash sale: the same edge throttle protects the seat inventory rows this quest calls the hottest in the database.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.


