Want it harder? Hide the model answer and work each phase in a scratchpad, then reveal and score yourself. Or simply read on: the checkpoints below work either way.
A Revolution That Changed a Nation
In 2016, India launched an experiment. Could a country of 1.4 billion people, many unbanked, leapfrog decades of payment infrastructure and go directly to real-time digital payments?
Ten years later, the answer is extraordinary.
- ~800 Million — daily transactions (802M average, September 2026)
- 24+ Billion — monthly transactions (24.07B, September 2026)
- 228 Billion — annual transactions (calendar 2025)
$3.5 Trillion (₹300 Lakh Crore) — annual value (calendar 2025)- 740+ — participating banks (741, July 2026)
- 550+ Million — users onboarded (June 2026)
- A few seconds — typical end-to-end payment time
- ~99% — technical success rate (technical declines at 0.7-0.8%, NPCI's CEO, November 2024)
- ~49% of the world's real-time payment volume — global share (2023 data, ACI Worldwide)
- 11 (Singapore, UAE, France, Nepal, Bhutan, Sri Lanka, Mauritius, Qatar...) — countries accepting UPI (August 2026)
For context
- UPI processes MORE transactions than Visa and Mastercard COMBINED in India
- A tea vendor in rural India uses the same system as a Fortune 500 company
- Transactions as small as ₹1 (about 1 US cent) work with the same reliability as ₹1 Crore
- The NPCI switch runs 24/7/365; individual banks still announce short maintenance windows (HDFC Bank did several in 2025)
This is the system we'll design today.
UPI moves about 800 million payments a day between 740+ banks in under 2 seconds: ~9,300 TPS on average, 28,000 to 46,000 at peak. Apps (PSPs) talk to the NPCI switch, which resolves the VPA, debits the payer's bank and credits the payee's while money settles bank to bank through the RBI. The deep dives cover cached VPA resolution, a state machine with DEEMED_SUCCESS and auto-reversal by T+1, per-bank circuit breakers and layered security, then a 5x Diwali, data-centre failover, idempotent retries and the April 2025 status-check outage.
The Interview Begins#
You're interviewing at a fintech company. The principal architect draws on the whiteboard:
"India's UPI is considered one of the greatest achievements in financial technology. Countries around the world are trying to replicate it. Today, I want you to design a system like UPI from scratch."
Build the infrastructure that enables instant money transfers between ANY two bank accounts in a country using just a phone.
Requirements
- Support 700+ banks with different legacy systems
- Handle 800+ million transactions per day
- Complete each transaction in < 2 seconds end-to-end
- 99.9% availability (< 9 hours downtime/year)
- Zero tolerance for money loss (atomic transactions)
- Work on basic smartphones with 2G/3G connectivity
- Support both P2P (person-to-person) and P2M (person-to-merchant)
"This is arguably one of the hardest system design problems — you're building financial infrastructure for a nation. Take your time."
Phase 1: Requirements Clarification#
"Before I start, let me understand the constraints better."
Your Questions#
"First, what's the relationship between banks and this central system? Do banks connect directly, or through intermediaries?"
"Banks connect through the central switch — NPCI in India's case. They don't talk to each other directly. The central system orchestrates everything."
"What about the mobile apps? Can any app connect to the system?"
"Apps must be approved and must partner with a bank. We call these Payment Service Providers or PSPs. PhonePe started with Yes Bank (hence @ybl), Google Pay works with multiple banks. The app itself doesn't hold money — it's just an interface."
"How do users identify each other? Bank account numbers are long and error-prone."
"Great observation. UPI solved this with Virtual Payment Addresses — like an email for money. username@bankname. The system maps this to actual bank accounts."
"What happens if a transaction fails mid-way? Say money is debited but not credited?"
"This is critical. The system MUST be atomic. Either the full transaction succeeds, or it's completely rolled back. Users cannot lose money due to technical failures."
"What's the peak traffic pattern? Is it bursty?"
"Very bursty. Evenings run well above the daily average, and festival days like Diwali set new records. Plan for peaks of 3-5x the average so the system handles them gracefully."
Requirements Summary#
Functional Requirements:
1. User management
- Register users via mobile number + bank account
- Create and manage Virtual Payment Addresses (VPAs)
- Link multiple bank accounts to one app
- Two-factor authentication (device binding + PIN)
2. Payment operations
- Push payments (I send money to you)
- Pull payments / Collect requests (I request money from you)
- QR code payments (scan and pay)
- Recurring payments (autopay/mandates)
3. Transaction processing
- Real-time debit from sender's bank
- Real-time credit to receiver's bank
- Transaction status tracking
- Refund/reversal handling
4. Bank integration
- Standardized APIs for all banks
- VPA to account resolution
- Balance inquiry (with consent)
- Account validation
5. Settlement
- Net settlement between banks (periodic)
- Reconciliation and dispute handling
- Audit trail for compliance
Non-Functional Requirements:
Scale
- 800+ million transactions/day
- 24+ billion transactions/month
- 550+ million users
- 740+ participating banks
Latency
- End-to-end: < 2 seconds (p99)
- NPCI switch processing: < 300ms
- Bank response time: < 700ms per call (debit and credit)
Availability
- 99.9% uptime (8.7 hours downtime/year max)
- 24/7/365 operation
- No scheduled maintenance windows
Consistency
- ACID transactions (money can't be lost)
- Exactly-once semantics
- Atomic debit-credit operations
Security
- End-to-end encryption
- Device binding
- Multi-factor authentication
- Fraud detection in real-time
Phase 2: Back of the Envelope Estimation#
"Let me work through the numbers..."
Before the numbers: what does 800 million a day mean per second?
Payments run at about 800 million a day, and you plan for evening and festival peaks of 3 to 5 times the average. Work out the average and the peak transactions per second. Then: every payment also needs a payee lookup, two bank calls, audit writes and two notifications. Roughly how many backend operations per second is the peak?
One strong answer
- Average: 800,000,000 ÷ 86,400 s ≈ 9,300 TPS.
- Peak: 3 to 5 times that, so ~28,000 to 46,000 TPS.
- Fan-out: each payment is about six operations (a lookup, a debit call, a credit call, one or two audit writes, two pushes), so the peak is ~170,000+ operations per second. That fan-out, not the raw payment count, is what you size the switch, the queues and the bank links for.
Traffic Calculations#
- Daily transactions: 800,000,000
- Seconds per day: 86,400
- Average TPS: ~9,300 TPS
- Peak multiplier: 3-5x (design headroom for evenings, festivals)
- Peak TPS: ~28,000-46,000 TPS
- Per transaction, multiple operations:
- VPA resolution: 1 lookup
- Sender bank call: 1 API call
- Receiver bank call: 1 API call
- Audit logging: 1-2 writes
- Notifications: 2 pushes
- Effective operations/second: ~170,000+ at peak
Data Volume#
- Per transaction record: ~500 bytes
- Transaction ID: 36 bytes (UUID)
- Sender VPA: 50 bytes
- Receiver VPA: 50 bytes
- Amount: 8 bytes
- Timestamps: 16 bytes
- Status: 4 bytes
- Bank references: 100 bytes
- Metadata: 200 bytes
- Daily storage: ~900 GB/day
- Transactions: 800M × 500B = 400 GB/day
- Audit logs: ~500 GB/day
- Annual storage: ~330 TB/year
- 7-year retention: ~2.3 PB
Infrastructure Estimates#
At ~28,000 TPS peak, assuming each server handles 1,000 TPS:
- API servers: 28+ servers (with redundancy: 56+)
- Database: Clustered, sharded
- Cache: Distributed Redis cluster
- Message queues: High-throughput Kafka cluster
Network
- Connections to 740+ banks
- Each bank: dedicated secure link
- Geographic distribution: Multiple data centers
- Bandwidth: Several Gbps
Requirements and estimation
Q1/3 With 800 million transactions a day and a 3-5x peak multiplier, what peak transaction rate should the switch be sized for?
Answer & explanation
Answer: B
800M / 86,400 s is about 9,300 TPS on average; 3-5x gives roughly 28,000-46,000 TPS at peak. The ~170,000 figure is effective operations per second, not transactions, so sizing transaction capacity on it overshoots.
Q2/3 Why do ~28,000-46,000 peak transactions per second turn into ~170,000+ operations per second?
Answer & explanation
Answer: C
One payment is not one operation: it needs a VPA resolution, a debit call, a credit call, 1-2 audit writes and 2 pushes. Retries happen, but they are not what multiplies the load in the estimate.
Q3/3 Storage comes to ~900 GB/day (transactions plus audit logs). What does 7-year retention need, roughly?
Answer & explanation
Answer: D
900 GB x 365 is roughly 330 TB a year, and 7 years of that is about 2.3 PB. 200 TB undercounts by forgetting the multi-year retention; 20 PB is an order of magnitude too high.
Phase 3: High-Level Architecture#
"Let me draw how UPI actually works. It's a beautiful layered architecture."
Sketch it first: who holds the money?
Hundreds of banks, dozens of apps, one central switch. Sketch the layers. Then decide: while a payment is in flight, should the central switch hold the money itself?
One strong answer
Three layers:
- Apps (PSPs): each app partners with a bank, onboards users and creates their payment addresses. The app never holds money.
- The central switch (NPCI): routes every request between banks and runs the transaction state machine.
- Banks: hold the accounts and post the debit and the credit.
The switch must not hold money. Information flows app → PSP → switch → banks, while money moves bank to bank: customer accounts are debited and credited in real time, and the interbank obligations are netted and settled in batches through the banks' accounts at the central bank. So the switch carries no counterparty risk, banks stay the regulated entities, and the switch can scale out like any stateless router plus a transaction log.
The Three-Layer Cake#
%% title: UPI architecture — the three-layer cake
flowchart TD
subgraph L1["Layer 1 · User interface"]
PP@{ icon: "sd:phone", label: "PhonePe (App)" }
GP@{ icon: "sd:phone", label: "Google Pay (App)" }
PT@{ icon: "sd:phone", label: "Paytm (App)" }
BH@{ icon: "sd:phone", label: "BHIM (App)" }
PSP["PSP (Payment Service Providers)<br/>User onboarding · VPA creation · UI/UX"]:::edge
end
PP & GP & PT & BH --> PSP
subgraph L2["Layer 2 · NPCI switch"]
subgraph NPCI["NPCI UPI platform"]
VPA["VPA Mapper"]:::service
TR["Transaction Router"]:::hot
FD["Fraud Detection"]:::service
SE["Settlement Engine"]:::service
AT[("Audit Trail")]:::store
DR["Dispute Resolution"]:::service
direction TB
VPA ~~~ SE
TR ~~~ AT
FD ~~~ DR
end
end
PSP ==>|"information"| NPCI
subgraph L3["Layer 3 · Banking layer"]
direction TB
SBI@{ icon: "sd:bank", label: "SBI Bank" }
HDFC@{ icon: "sd:bank", label: "HDFC Bank" }
ICICI@{ icon: "sd:bank", label: "ICICI Bank" }
AXIS@{ icon: "sd:bank", label: "Axis Bank" }
MORE@{ icon: "sd:bank", label: "740+ Banks" }
IMPS[["IMPS rails + RBI net settlement<br/>how money moves"]]:::queue
end
NPCI ==>|"information"| L3
SBI & HDFC & ICICI & AXIS & MORE -->|"money"| IMPS
class PP,GP,PT,BH client
class SBI,HDFC,ICICI,AXIS,MORE external- PSP (Payment Service Providers): apps must partner with a bank (PSP Bank) to access UPI — PhonePe → Yes Bank, Google Pay → multiple banks. The PSP handles user onboarding, VPA creation and UI/UX.
- NPCI UPI platform: the brain of UPI — routes transactions between banks. NPCI NEVER holds money — only routes information.
- IMPS (Immediate Payment Service) and settlement: UPI is built on the IMPS rails — customer accounts are debited and credited in real time, while the interbank obligations are netted by NPCI and settled in batches (several cycles a day) through the banks' accounts at the RBI.
KEY INSIGHT:
- Money flows: Bank → Bank (real-time account postings; net interbank settlement via RBI)
- Information flows: App → PSP → NPCI → Banks
- NPCI is the orchestrator, not a money holder
Transaction Flow#
"Let me trace a ₹500 payment from Alice to Bob..."
%% title: Transaction flow — Alice pays Bob ₹500
sequenceDiagram
autonumber
participant A as Alice's phone (PhonePe app)
participant N as NPCI switch
box Banks
participant H as HDFC (Alice's bank)
participant X as Axis (Bob's bank)
end
A->>N: INITIATE · "Pay ₹500 to bob@okaxis"
Note over A,N: + Alice's VPA<br/>+ Encrypted PIN
N->>X: RESOLVE VPA · "Who is bob@okaxis?"
X-->>N: VPA RESPONSE · "Bob's account at Axis Bank: XXXX1234"
N->>H: DEBIT REQUEST · "Debit ₹500 from Alice at HDFC"
H-->>N: DEBIT RESPONSE · "Debited. Ref: ABC123"
N->>X: CREDIT REQUEST · "Credit ₹500 to Bob at Axis"
X-->>N: CREDIT RESPONSE · "Credited. Ref: XYZ789"
N-->>A: SUCCESS · "Payment complete! Ref: TXN123456"
Note over A,X: TOTAL TIME: < 2 secondsPARALLEL ACTIONS:
- Audit log written at each step
- Fraud check runs during step ①
- Push notifications sent to both Alice and Bob
- Settlement record created for bank reconciliation
High-level architecture
Q1/2 Alice pays a merchant through PhonePe. Where does her money actually move?
Answer & explanation
Answer: A
Money flows bank to bank (real-time postings, net interbank settlement via RBI); information flows App -> PSP -> NPCI -> Banks. The PSP app is only an interface and NPCI is an orchestrator that never holds money, so neither is a hop for the funds.
Q2/2 In the Alice-to-Bob flow, why does NPCI resolve
bob@okaxisbefore sending the debit to HDFC?Answer & explanation
Answer: C
The credit leg needs the real account behind the VPA, and resolving first means an invalid VPA fails cleanly with no debit to undo. Fraud checks run in parallel at initiation; they are not what VPA resolution is for.
Phase 4: Deep Dives#
Deep Dive 1: Virtual Payment Address (VPA) Resolution#
Week 1 concepts: Partitioning, lookup optimization. Week 4 concepts: Caching.
"VPA resolution is called for EVERY transaction. With 800 million daily transactions, this lookup must be blazing fast."
The Challenge:
Hundreds of millions of VPAs like
alice@okhdfcbankbob@okaxismerchant@paytm9876543210@ybl- ...
Each VPA maps to
- Bank code
- Account number (encrypted)
- Account holder name
- Status (active/blocked)
- Metadata
Requirements
- Lookup latency: < 10ms
- 100% accuracy (wrong mapping = money to wrong person!)
- Real-time updates (user changes bank)
- Handle 50,000+ lookups/second at peak
The Solution:
%% title: VPA mapper architecture — VPA resolution flow
flowchart TD
IN(["VPA: bob@okaxis"]):::client
P["PARSE HANDLE<br/>handle='bob' · suffix='okaxis'"]:::service
S["SUFFIX LOOKUP<br/>'okaxis' → Axis Bank Code"]:::core
BR@{ icon: "sd:database", label: "BANK REGISTRY<br/>okhdfcbank → HDFC · okaxis → Axis<br/>oksbi → SBI · ybl → Yes Bank" }
C{"CACHE CHECK (Redis)<br/>Cache hit?"}:::cache
Q["QUERY BANK (Real-time)"]:::service
AX@{ icon: "sd:bank", label: "Axis Bank<br/>VPA Database" }
RET(["RETURN ACCOUNT DETAILS<br/>Bank: Axis · Account: ***234<br/>Name: Bob Kumar"]):::ok
IN --> P --> S
S --> BR
S --> C
C ==>|"Yes"| RET
C -->|"No"| Q
Q --> AX
Q -->|"Update cache"| RET
class BR store
class AX externalKEY DESIGN DECISIONS (a likely design; NPCI doesn't publish its switch internals):
- DISTRIBUTED OWNERSHIP — Each bank owns their VPA namespace (suffix). NPCI doesn't store all VPAs — banks do. This scales naturally as banks handle their own data.
- CACHING STRATEGY — Hot VPAs (frequently used) cached at NPCI. TTL: 15-30 minutes. Invalidation: Banks push updates for VPA changes.
- HANDLE UNIQUENESS — handle@suffix is globally unique. Banks ensure uniqueness within their namespace. Cross-bank uniqueness handled by suffix differentiation.
# vpa/resolver.py
"""
VPA Resolution Service
Maps Virtual Payment Addresses to actual bank accounts.
This is the most critical lookup in the entire system.
"""
from dataclasses import dataclass
from typing import Optional, Tuple
from datetime import datetime, timedelta
import hashlib
@dataclass
class AccountDetails:
"""Resolved account information."""
bank_code: str
account_number_masked: str # Only last 4 digits visible
account_holder_name: str
ifsc_code: str
is_active: bool
verified_at: datetime
@dataclass
class VPAResolutionResult:
"""Result of VPA resolution."""
success: bool
account: Optional[AccountDetails] = None
error_code: Optional[str] = None
resolution_time_ms: float = 0
cache_hit: bool = False
class VPAResolver:
"""
Resolves VPAs to bank account details.
Design principles:
- Cache aggressively (VPAs don't change often)
- Fail fast on invalid formats
- Banks are the source of truth
"""
def __init__(
self,
cache, # Redis cluster
bank_registry, # Bank code → Bank API mapping
bank_gateway, # Gateway to call bank APIs
metrics
):
self.cache = cache
self.registry = bank_registry
self.gateway = bank_gateway
self.metrics = metrics
# Cache settings
self.cache_ttl = timedelta(minutes=30)
self.negative_cache_ttl = timedelta(minutes=5)
async def resolve(self, vpa: str) -> VPAResolutionResult:
"""
Resolve a VPA to account details.
VPA format: handle@suffix
Example: alice@okhdfcbank, 9876543210@ybl
"""
start_time = datetime.utcnow()
# Step 1: Parse and validate VPA format
parsed = self._parse_vpa(vpa)
if not parsed:
return VPAResolutionResult(
success=False,
error_code="INVALID_VPA_FORMAT"
)
handle, suffix = parsed
# Step 2: Get bank code from suffix
bank_code = self.registry.get_bank_for_suffix(suffix)
if not bank_code:
return VPAResolutionResult(
success=False,
error_code="UNKNOWN_VPA_SUFFIX"
)
# Step 3: Check cache
cache_key = f"vpa:{vpa.lower()}"
cached = await self.cache.get(cache_key)
if cached:
if cached == "NOT_FOUND":
return VPAResolutionResult(
success=False,
error_code="VPA_NOT_FOUND",
cache_hit=True
)
account = AccountDetails(**cached)
return VPAResolutionResult(
success=True,
account=account,
resolution_time_ms=self._elapsed_ms(start_time),
cache_hit=True
)
# Step 4: Query the bank
try:
account = await self.gateway.resolve_vpa(
bank_code=bank_code,
handle=handle,
suffix=suffix
)
if account:
# Cache the result
await self.cache.set(
cache_key,
account.__dict__,
ttl=self.cache_ttl
)
return VPAResolutionResult(
success=True,
account=account,
resolution_time_ms=self._elapsed_ms(start_time),
cache_hit=False
)
else:
# Cache negative result (VPA doesn't exist)
await self.cache.set(
cache_key,
"NOT_FOUND",
ttl=self.negative_cache_ttl
)
return VPAResolutionResult(
success=False,
error_code="VPA_NOT_FOUND",
resolution_time_ms=self._elapsed_ms(start_time)
)
except BankTimeoutError:
return VPAResolutionResult(
success=False,
error_code="BANK_TIMEOUT"
)
except BankUnavailableError:
return VPAResolutionResult(
success=False,
error_code="BANK_UNAVAILABLE"
)
def _parse_vpa(self, vpa: str) -> Optional[Tuple[str, str]]:
"""Parse VPA into handle and suffix."""
if not vpa or '@' not in vpa:
return None
parts = vpa.lower().strip().split('@')
if len(parts) != 2:
return None
handle, suffix = parts
# Validate handle (alphanumeric, 3-50 chars)
if not handle or len(handle) < 3 or len(handle) > 50:
return None
# Validate suffix (registered bank suffix)
if not suffix or len(suffix) < 2 or len(suffix) > 20:
return None
return handle, suffix
def _elapsed_ms(self, start: datetime) -> float:
return (datetime.utcnow() - start).total_seconds() * 1000Deep Dive 2: Atomic Transactions — The Heart of Trust#
Week 2 concepts: Idempotency, failure handling. Week 5 concepts: Distributed transactions, Saga pattern.
"The most critical requirement: money cannot disappear. If I debit Alice but fail to credit Bob, Alice must get her money back. Always."
The credit call just timed out
Alice's bank has debited ₹500. The call to credit Bob's bank timed out, so you don't know whether Bob was credited. What do you do right now, and how do you guarantee the money is never lost and never moved twice?
One strong answer
- Don't guess. Re-sending the debit risks a double debit, and reversing at once is wrong if the credit actually landed.
- Record the uncertainty: the transaction moves to an in-between state (UPI calls it deemed success) and goes into the settlement file.
- Resolve it with the same ID: status checks and retries carry the original transaction ID, so every bank call is idempotent.
- Reconcile: the banks compare the settlement file with their actual credits. If the credit happened, the payment completes. If it didn't, the debit is reversed automatically, by T+1 at the latest under the RBI's turnaround rules.
The guarantee rests on one transaction ID end to end, an explicit state machine, and reconciliation as the backstop.
The Challenge:
Happy path
- Debit Alice (HDFC): ₹500 — ok
- Credit Bob (Axis): ₹500 — ok
→ Success!
Failure scenarios
Scenario A: Credit fails after debit
- Debit Alice: ₹500 — ok (money left Alice's account)
- Credit Bob: TIMEOUT — no (did it go through or not?)
→ UNCERTAINTY! Alice lost ₹500?
Scenario B: Network partition
- Debit Alice: ok
- Credit Bob: Request sent...
- Network dies
- We don't know the outcome!
→ UNCERTAINTY!
Scenario C: Duplicate request
- User clicks "Pay" twice quickly
- Two debit requests sent
→ DOUBLE DEBIT! Alice loses ₹1000?
These scenarios CANNOT happen in a payment system.
The Solution:
%% title: A likely transaction state machine (deemed success is UPI's real term; the other states are this design's) stateDiagram-v2 [*] --> CREATED CREATED --> PENDING: Validation passed PENDING --> DEBIT_INITIATED: Send to remitter bank DEBIT_INITIATED --> FAILED DEBIT_INITIATED --> DEBITED DEBITED --> CREDIT_INITIATED: Send to beneficiary bank CREDIT_INITIATED --> COMPLETED CREDIT_INITIATED --> REVERSED CREDIT_INITIATED --> DEEMED_SUCCESS: Credit timeout/failure DEEMED_SUCCESS --> REVERSED: Auto-reverse by T+1 DEBIT_INITIATED: DEBIT INITIATED CREDIT_INITIATED: CREDIT INITIATED FAILED: FAILED · no debit happened DEBITED: DEBITED · money left sender COMPLETED: COMPLETED · money received REVERSED: REVERSED · money back to sender DEEMED_SUCCESS: DEEMED SUCCESS · uncertain state class CREATED,PENDING,DEBIT_INITIATED,DEBITED,CREDIT_INITIATED core class COMPLETED ok class FAILED,REVERSED bad class DEEMED_SUCCESS queue
DEEMED SUCCESS: Bank didn't respond in time. Settlement happens; if credit actually failed, it must be auto-reversed by T+1.
Alice pays ₹500 to bob@okaxis: NPCI resolves the VPA at Axis, debits Alice at HDFC (ref ABC123), credits Bob at Axis (ref XYZ789) and confirms in under 2 seconds, routing only messages while the money moves bank to bank. If the credit call times out, NPCI marks the payment DEEMED_SUCCESS and puts it in the settlement file; when reconciliation finds no credit, the debit is auto-reversed by T+1 with the same transaction ID.
# transaction/processor.py
"""
Transaction Processing with Atomic Guarantees.
This is the most critical code in the entire system.
Money cannot be lost under any circumstances.
"""
from dataclasses import dataclass
from typing import Optional
from enum import Enum
from datetime import datetime, timedelta
import uuid
class TransactionState(Enum):
CREATED = "created"
PENDING = "pending"
DEBIT_INITIATED = "debit_initiated"
DEBITED = "debited"
CREDIT_INITIATED = "credit_initiated"
COMPLETED = "completed"
FAILED = "failed"
DEEMED_SUCCESS = "deemed_success"
REVERSED = "reversed"
@dataclass
class Transaction:
"""A UPI transaction record."""
txn_id: str
sender_vpa: str
receiver_vpa: str
amount: int # In paise (smallest unit)
state: TransactionState
created_at: datetime
updated_at: datetime
# Bank references
sender_bank_ref: Optional[str] = None
receiver_bank_ref: Optional[str] = None
# For idempotency
idempotency_key: Optional[str] = None
# Reversal tracking
reversal_initiated: bool = False
reversal_completed: bool = False
class TransactionProcessor:
"""
Processes UPI transactions with atomic guarantees.
Key principles:
1. IDEMPOTENCY: Same request = same result (no double-debit)
2. ATOMICITY: Either complete success or complete rollback
3. DURABILITY: State persisted before any bank call
4. RECOVERABILITY: Can resume from any failure point
"""
def __init__(
self,
db, # Transaction database
bank_gateway, # Bank API gateway
reversal_queue, # Queue for failed transactions
audit_log
):
self.db = db
self.gateway = bank_gateway
self.reversal_queue = reversal_queue
self.audit = audit_log
# Timeouts
self.debit_timeout = timedelta(seconds=30)
self.credit_timeout = timedelta(seconds=30)
async def process(
self,
sender_vpa: str,
receiver_vpa: str,
amount: int,
idempotency_key: str
) -> Transaction:
"""
Process a payment transaction.
CRITICAL: This method must be idempotent.
Same idempotency_key = same result, always.
"""
# STEP 0: Check idempotency
existing = await self.db.get_by_idempotency_key(idempotency_key)
if existing:
# Return existing result (no reprocessing)
await self.audit.log("IDEMPOTENT_RETURN", existing.txn_id)
return existing
# STEP 1: Create transaction record FIRST
txn = Transaction(
txn_id=str(uuid.uuid4()),
sender_vpa=sender_vpa,
receiver_vpa=receiver_vpa,
amount=amount,
state=TransactionState.CREATED,
created_at=datetime.utcnow(),
updated_at=datetime.utcnow(),
idempotency_key=idempotency_key
)
# Persist BEFORE any bank call
await self.db.save(txn)
await self.audit.log("TXN_CREATED", txn.txn_id)
try:
# STEP 2: Initiate debit
txn.state = TransactionState.DEBIT_INITIATED
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
debit_result = await self.gateway.debit(
vpa=sender_vpa,
amount=amount,
txn_ref=txn.txn_id,
timeout=self.debit_timeout
)
if not debit_result.success:
# Debit failed cleanly — no money moved
txn.state = TransactionState.FAILED
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
await self.audit.log("DEBIT_FAILED", txn.txn_id,
debit_result.error)
return txn
# STEP 3: Debit succeeded — record it
txn.state = TransactionState.DEBITED
txn.sender_bank_ref = debit_result.bank_reference
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
await self.audit.log("DEBIT_SUCCESS", txn.txn_id)
# STEP 4: Initiate credit
# CRITICAL: From this point, we MUST either complete or reverse
txn.state = TransactionState.CREDIT_INITIATED
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
credit_result = await self.gateway.credit(
vpa=receiver_vpa,
amount=amount,
txn_ref=txn.txn_id,
timeout=self.credit_timeout
)
if credit_result.success:
# SUCCESS! Transaction complete
txn.state = TransactionState.COMPLETED
txn.receiver_bank_ref = credit_result.bank_reference
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
await self.audit.log("TXN_COMPLETED", txn.txn_id)
return txn
elif credit_result.status == "TIMEOUT":
# UNCERTAINTY: We don't know if credit happened
# Mark as DEEMED_SUCCESS — settlement will clarify
txn.state = TransactionState.DEEMED_SUCCESS
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
await self.audit.log("TXN_DEEMED_SUCCESS", txn.txn_id)
# Schedule reconciliation check
await self.reversal_queue.schedule_check(
txn.txn_id,
check_at=datetime.utcnow() + timedelta(hours=24)
)
return txn
else:
# Credit FAILED — must reverse the debit
await self._initiate_reversal(txn)
return txn
except Exception as e:
# Unexpected error — check state and recover
await self.audit.log("TXN_ERROR", txn.txn_id, str(e))
await self._handle_error(txn, e)
raise
async def _initiate_reversal(self, txn: Transaction):
"""
Reverse a failed transaction.
Credit the debited amount back to sender.
"""
txn.reversal_initiated = True
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
await self.audit.log("REVERSAL_INITIATED", txn.txn_id)
# Queue for reversal (handled by separate process)
await self.reversal_queue.enqueue(txn.txn_id)
async def process_reversal(self, txn_id: str):
"""
Execute reversal — credit money back to sender.
Called by reversal worker.
"""
txn = await self.db.get(txn_id)
if txn.reversal_completed:
return # Already reversed
# Credit back to sender
reversal_result = await self.gateway.credit(
vpa=txn.sender_vpa,
amount=txn.amount,
txn_ref=f"REV-{txn.txn_id}",
timeout=self.credit_timeout
)
if reversal_result.success:
txn.state = TransactionState.REVERSED
txn.reversal_completed = True
txn.updated_at = datetime.utcnow()
await self.db.save(txn)
await self.audit.log("REVERSAL_COMPLETED", txn.txn_id)
else:
# Reversal failed — retry later
# This is a critical alert scenario
await self.audit.log("REVERSAL_FAILED", txn.txn_id,
reversal_result.error)
await self.reversal_queue.schedule_retry(
txn.txn_id,
retry_at=datetime.utcnow() + timedelta(minutes=15)
)"What about the 'deemed success' state? That seems risky."
"Great catch. Here's how reconciliation handles it..."
Scenario: We debited Alice, tried to credit Bob, got TIMEOUT
At NPCI level
- Transaction marked
DEEMED_SUCCESS - We don't know if Bob got money
- Settlement file sent to banks includes this transaction
At Bank level (reconciliation, by T+1)
- Bank compares settlement file with actual credits
- If credit happened: Mark as COMPLETED
- If credit NOT happened: Mark as FAILED → Auto-reversal
Timing
- T+0 — Transaction happens, deemed success
- T+0/T+1 — Banks reconcile, report actual status
- By end of T+1 — If the credit failed, the debit is reversed automatically
This is why the RBI's turnaround-time rules say that for a UPI account-to-account transfer where the account is debited but the beneficiary is not credited, the reversal must happen by T+1; after that the bank owes the customer ₹100 per day of delay (for merchant payments the limit is T+5).
In practice, most such cases resolve within hours.
VPA resolution and atomic transactions
Q1/3 The resolver caches a found VPA for 30 minutes but a "not found" result for only 5 minutes. Why the shorter TTL for misses?
Answer & explanation
Answer: B
Caching "NOT_FOUND" stops repeated lookups of a bad VPA from hitting the bank, while the short TTL limits how long a just-registered VPA looks missing. Timeouts are returned as BANK_TIMEOUT and are not cached as misses at all.
Q2/3 Alice is debited, then the credit call to Bob's bank times out. What should the processor do?
Answer & explanation
Answer: D
A timeout means the credit may or may not have happened. DEEMED_SUCCESS records that uncertainty, and bank reconciliation later confirms it or triggers auto-reversal. Reversing right away is the tempting mistake: if Bob was in fact credited, money would be created out of thin air.
Q3/3 Why does the processor save the transaction record (CREATED, then DEBIT_INITIATED) before calling any bank?
Answer & explanation
Answer: A
Durability before side effects is what makes recovery possible: the system always knows how far a payment got and whether money must be put back. It does not speed anything up; it adds a write, and that cost is accepted for recoverability.
Deep Dive 3: Bank Integration at Scale#
Week 2 concepts: Timeouts, circuit breakers. Week 3 concepts: Message queues.
"With 740+ banks, each with different legacy systems, bank integration is a massive challenge."
%% title: Bank integration architecture — NPCI → bank gateway
flowchart TD
subgraph GW["NPCI → Bank gateway"]
direction TB
subgraph AL["Bank adapter layer"]
SA["SBI Adapter<br/>SBI's quirks · retry logic · timeouts"]:::service
HA["HDFC Adapter<br/>HDFC's quirks · retry logic · timeouts"]:::service
XA["Axis Adapter<br/>Axis's quirks · retry logic · timeouts"]:::service
MA["..."]:::service
end
subgraph CB["Circuit breaker layer · per-bank breakers prevent cascade failures"]
CS["SBI: CLOSED · healthy"]:::ok
CH["HDFC: CLOSED · healthy"]:::ok
CX["Axis: OPEN · failing, skip for 30s"]:::bad
CI["ICICI: HALF · testing recovery"]:::queue
direction TB
CS ~~~ CX
CH ~~~ CI
end
SEC["SECURE COMMUNICATION LAYER<br/>HTTPS with mutual TLS<br/>Request/Response signing<br/>Encryption of sensitive data<br/>IP whitelisting<br/>Dedicated leased lines to major banks"]:::edge
end
SA & HA & XA --> CB
CB --> SECEvery bank implements NPCI's standard UPI API (XML messages), but their latency, reliability and error behaviour vary widely. In this design, a per-bank adapter holds each bank's timeouts and retry rules; the format quirks in the code below are illustrative of a generic multi-bank gateway, not of UPI.
# bank/gateway.py
"""
Bank Gateway: Unified interface to 740+ banks.
Each bank is different. This gateway provides a
consistent interface while handling per-bank quirks.
"""
from dataclasses import dataclass
from typing import Dict, Optional
from datetime import datetime, timedelta
from enum import Enum
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Bank failing, don't try
HALF_OPEN = "half" # Testing if bank recovered
@dataclass
class BankConfig:
"""Configuration for a bank."""
bank_code: str
endpoint: str
timeout_ms: int = 30000
# Circuit breaker settings
failure_threshold: int = 5
recovery_timeout_s: int = 30
# Illustrative quirks for a generic multi-bank gateway
# (UPI itself gives every bank one standard XML API)
requires_padding: bool = False
amount_in_rupees: bool = False # e.g. a partner that wants rupees, not paise
legacy_xml_format: bool = False
class BankGateway:
"""
Gateway for all bank operations.
"""
def __init__(
self,
http_client,
bank_configs: Dict[str, BankConfig],
metrics
):
self.http = http_client
self.configs = bank_configs
self.metrics = metrics
# Circuit breakers per bank
self.circuits: Dict[str, CircuitBreaker] = {
code: CircuitBreaker(config)
for code, config in bank_configs.items()
}
async def debit(
self,
bank_code: str,
account_ref: str,
amount: int,
txn_ref: str
) -> 'BankResponse':
"""
Debit an account at a bank.
Amount is in paise (smallest unit).
"""
config = self.configs.get(bank_code)
if not config:
return BankResponse(
success=False,
error_code="UNKNOWN_BANK"
)
# Check circuit breaker
circuit = self.circuits[bank_code]
if not circuit.can_execute():
self.metrics.increment("bank_circuit_open", bank_code)
return BankResponse(
success=False,
error_code="BANK_CIRCUIT_OPEN"
)
try:
# Build request (handle bank-specific formats)
request = self._build_debit_request(
config, account_ref, amount, txn_ref
)
# Make the call
start = datetime.utcnow()
response = await self.http.post(
config.endpoint + "/debit",
json=request,
timeout=config.timeout_ms / 1000
)
latency = (datetime.utcnow() - start).total_seconds() * 1000
# Record metrics
self.metrics.record_latency("bank_debit", bank_code, latency)
# Parse response
result = self._parse_response(config, response)
if result.success:
circuit.record_success()
else:
circuit.record_failure()
return result
except TimeoutError:
circuit.record_failure()
self.metrics.increment("bank_timeout", bank_code)
return BankResponse(
success=False,
error_code="TIMEOUT",
status="TIMEOUT"
)
except Exception as e:
circuit.record_failure()
self.metrics.increment("bank_error", bank_code)
return BankResponse(
success=False,
error_code="BANK_ERROR",
error_message=str(e)
)
def _build_debit_request(
self,
config: BankConfig,
account_ref: str,
amount: int,
txn_ref: str
) -> dict:
"""Build bank-specific request format."""
# Handle amount format (paise vs rupees)
if config.amount_in_rupees:
amount_value = amount / 100
else:
amount_value = amount
if config.legacy_xml_format:
# Illustrative: a partner on an older format
return {
"xml_payload": self._build_xml(
account_ref, amount_value, txn_ref
)
}
return {
"account_reference": account_ref,
"amount": amount_value,
"transaction_reference": txn_ref,
"timestamp": datetime.utcnow().isoformat()
}
class CircuitBreaker:
"""
Circuit breaker for bank connections.
Prevents cascade failures when a bank is down.
"""
def __init__(self, config: BankConfig):
self.config = config
self.state = CircuitState.CLOSED
self.failure_count = 0
self.last_failure: Optional[datetime] = None
self.success_count = 0
def can_execute(self) -> bool:
"""Check if we can make a request to this bank."""
if self.state == CircuitState.CLOSED:
return True
if self.state == CircuitState.OPEN:
# Check if recovery timeout passed
if self.last_failure:
elapsed = (datetime.utcnow() - self.last_failure).total_seconds()
if elapsed > self.config.recovery_timeout_s:
self.state = CircuitState.HALF_OPEN
self.success_count = 0
return True
return False
if self.state == CircuitState.HALF_OPEN:
return True
return False
def record_success(self):
"""Record a successful request."""
if self.state == CircuitState.HALF_OPEN:
self.success_count += 1
if self.success_count >= 3: # 3 successes to close
self.state = CircuitState.CLOSED
self.failure_count = 0
else:
self.failure_count = 0
def record_failure(self):
"""Record a failed request."""
self.failure_count += 1
self.last_failure = datetime.utcnow()
if self.state == CircuitState.HALF_OPEN:
# Back to open
self.state = CircuitState.OPEN
self.success_count = 0
elif self.failure_count >= self.config.failure_threshold:
self.state = CircuitState.OPENDeep Dive 4: Security — The Trust Foundation#
Week 9 concepts: Security, authentication, fraud detection.
"UPI handles nearly ₹30 trillion a month (₹29.37 lakh crore in September 2026). Security isn't optional — it's existential."
Multi-layer security
Layer 1: Device binding
- The app's UPI registration is bound to the device and SIM (the UPI PIN itself belongs to the bank account)
- Device identifier (a likely design detail; modern Android no longer exposes the IMEI to apps)
- SIM binding (mobile number verification)
- If device changes, re-registration required
Layer 2: Two-factor authentication
Factor 1: Something you HAVE
- The registered mobile device
- The SIM card with registered number
Factor 2: Something you KNOW
- 4-6 digit UPI PIN (set by user)
- PIN encrypted on device, never transmitted in clear
Layer 3: Encryption
- HTTPS/TLS for all communication
- UPI PIN encrypted on the device with a public key (asymmetric RSA, via NPCI's common library); hashing schemes like PBKDF2 do not encrypt
- PIN verification in bank's HSM (Hardware Security Module)
- End-to-end encryption for sensitive data
Layer 4: Transaction signing
- Each transaction signed with digital signature
- Prevents tampering in transit
- Non-repudiation for disputes
Layer 5: Real-time fraud detection
- Velocity checks (too many transactions too fast)
- Amount anomaly detection
- Geo-location checks (impossible travel)
- Behavioral analysis
- Block list matching
Infrastructure security
NPCI data centers
- Tier-IV design (Hyderabad is rated for 99.995% availability)
- Chennai + Hyderabad (geographically separated)
- HSMs for key storage (FIPS 140-2 Level 3 is the usual bar)
- Active-Active configuration (this design)
- N+N redundancy (NPCI, Hyderabad data centre, 2020)
- Physical security:
- Biometric access control
- 24/7 security personnel
- CCTV surveillance
- Man-trap entries
Bank connections
- Dedicated leased lines (not public internet)
- Mutual TLS authentication
- IP whitelisting
- Regular security audits
# security/fraud_detector.py
"""
Real-Time Fraud Detection for UPI.
Must decide in < 50ms whether to allow a transaction.
"""
from dataclasses import dataclass
from typing import List, Tuple
from datetime import datetime, timedelta
@dataclass
class FraudSignals:
"""Signals used for fraud detection."""
user_id: str
device_id: str
amount: int
receiver_vpa: str
# Velocity
txn_count_1hr: int
txn_count_24hr: int
total_amount_24hr: int
# Device
is_new_device: bool
device_age_days: int
# Behavioral
is_new_receiver: bool
typical_amount: int
typical_time_of_day: List[int]
current_hour: int
# Location
device_location: str
usual_locations: List[str]
class FraudDetector:
"""
Real-time fraud detection.
Must be FAST (< 50ms) and ACCURATE (low false positives).
"""
def __init__(self, ml_model, rules_engine, blocklist):
self.model = ml_model
self.rules = rules_engine
self.blocklist = blocklist
async def evaluate(
self,
signals: FraudSignals
) -> Tuple[str, float, List[str]]:
"""
Evaluate fraud risk.
Returns: (decision, confidence, triggered_rules)
decision: "ALLOW", "BLOCK", "STEP_UP"
"""
triggered_rules = []
# RULE 1: Blocklist check (instant)
if await self.blocklist.is_blocked(signals.user_id):
return "BLOCK", 1.0, ["USER_BLOCKED"]
if await self.blocklist.is_blocked(signals.device_id):
return "BLOCK", 1.0, ["DEVICE_BLOCKED"]
# RULE 2: Velocity checks
if signals.txn_count_1hr > 10:
triggered_rules.append("HIGH_VELOCITY_1HR")
if signals.txn_count_24hr > 50:
triggered_rules.append("HIGH_VELOCITY_24HR")
if signals.total_amount_24hr > 100000_00: # ₹1 lakh in paise
triggered_rules.append("HIGH_AMOUNT_24HR")
# RULE 3: Amount anomaly
if signals.amount > signals.typical_amount * 10:
triggered_rules.append("AMOUNT_ANOMALY")
# RULE 4: New device
if signals.is_new_device:
triggered_rules.append("NEW_DEVICE")
if signals.amount > 10000_00: # > ₹10,000 on new device
triggered_rules.append("HIGH_AMOUNT_NEW_DEVICE")
# RULE 5: Unusual time
if signals.current_hour not in signals.typical_time_of_day:
triggered_rules.append("UNUSUAL_TIME")
# RULE 6: Location check
if signals.device_location not in signals.usual_locations:
triggered_rules.append("UNUSUAL_LOCATION")
# ML model for complex patterns
ml_score = await self.model.predict(signals)
# Decision logic
if ml_score > 0.9 or len(triggered_rules) > 3:
return "BLOCK", ml_score, triggered_rules
if ml_score > 0.7 or len(triggered_rules) > 1:
# Step-up: require additional verification
return "STEP_UP", ml_score, triggered_rules
return "ALLOW", 1 - ml_score, triggered_rulesBank integration and security
Q1/3 Axis Bank starts hanging on every request. Without a per-bank circuit breaker, what is the biggest risk?
Answer & explanation
Answer: D
Each hung call holds resources until its timeout, so one sick bank can drag down traffic for every bank. An OPEN breaker fails Axis calls fast instead. Double charging is prevented by idempotency, not by the breaker.
Q2/3 ICICI's breaker is HALF_OPEN. Two trial requests succeed, then the third fails. With this gateway code, what happens?
Answer & explanation
Answer: B
In HALF_OPEN any failure sends the breaker back to OPEN; it needs 3 successes in a row to close. The 5-failure threshold applies only when tripping from CLOSED, not while testing recovery.
Q3/3 On a new device, a user sends more than ₹10,000. Only NEW_DEVICE and HIGH_AMOUNT_NEW_DEVICE fire, and the ML score is 0.4. What does the fraud detector return?
Answer & explanation
Answer: C
Two triggered rules exceed the "more than 1" threshold for STEP_UP, even with a low ML score. BLOCK needs a score above 0.9 or more than 3 rules. A low ML score alone does not override the rule count.
Phase 5: Scaling and Edge Cases#
"What happens during Diwali when everyone is sending money?"
"Let's design for 5x spikes during festivals. Here's how..."
Diwali evening, and the largest bank starts timing out
Traffic is five times normal, packed into 7 to 11 PM, and the biggest bank's latency starts climbing. What did you prepare weeks before? And what stops one slow bank from dragging every other payment down with it?
One strong answer
- Before: pre-scale to around six times the normal peak, warm the VPA caches, tell banks to scale their own systems, and rehearse the runbooks.
- During: shed what can wait (promotional notifications), rate-limit per user, and queue non-urgent work such as mandate registrations.
- Isolation: a per-bank circuit breaker. When that bank's failures cross the threshold, payments involving it fail fast with a clear "bank unavailable" message, instead of every request waiting out a timeout and holding threads that other banks' payments need.
- And rate-limit every API, status checks included. Uncontrolled status-check traffic is what degraded the whole system in April 2025.
Festival Traffic Management#
| Normal day | Diwali | |
|---|---|---|
| Transactions | ~800 million | ~4 billion |
| Average TPS | ~9,300 | ~46,000 |
| Peak TPS | ~28,000 | ~140,000+ |
Diwali: Concentrated in evening hours (7 PM - 11 PM)
For reference, the real Diwali-eve 2025 peak was ~754 million transactions in a day, about 13% above October 2025's daily average of 668 million. The 5x here is deliberate headroom, not a forecast.
Preparation (weeks before)
- Pre-scale infrastructure to ~6x normal peak capacity (≈170K TPS)
- Warm up caches with popular VPAs
- Notify banks to scale their systems
- Extended support staff on standby
- Runbooks reviewed and tested
During the event
- Auto-scaling triggers at 60% capacity
- Non-critical features disabled (promotional notifications)
- Enhanced monitoring (5-second alert intervals)
- War room with all bank representatives
- Direct escalation paths to bank CTOs
Graceful degradation
If overwhelmed:
- Prioritize smaller transactions (more users served)
- Rate limit per-user (max 5 txn/minute)
- Queue non-urgent operations (mandate registrations)
- Return "Try again in few minutes" vs hard failure
Critical Edge Cases#
Edge case 1: Bank System Down
Problem: SBI (the largest remitter bank, roughly a quarter of all UPI debits: 5.2 billion of 20.0 billion in August 2025, per NPCI data) goes down
Impact: Roughly a quarter of transactions fail
Solution:
- Circuit breaker opens for SBI immediately
- Return clear error: "SBI temporarily unavailable"
- Pending transactions queued (if bank supports retry)
- Status page updated
- Auto-retry when circuit closes
- Transactions involving SBI gracefully rejected
Edge case 2: NPCI Switch Partial Failure
Problem: One NPCI data center fails
Impact: 50% capacity lost
Solution:
- Active-Active setup in Chennai and Hyderabad (this design; NPCI runs data centres in both cities)
- Traffic automatically routes to healthy DC
- Health-checked load balancing reroutes traffic (any DNS TTL kept ≤ 10-20 s, or a 60 s TTL alone would break the 30 s RTO)
- Data replicated synchronously between DCs
- RPO: 0 (no data loss), RTO: < 30 seconds
Edge case 3: Duplicate Transaction Request
Problem: User's app times out, they retry, but first request succeeded
Impact: Double debit
Solution:
- Every transaction has idempotency key
- Generated once when the payment is created (a unique transaction ID stored on the device/PSP) and reused verbatim on every retry; a key built from the retry's own timestamp would never match
- NPCI checks idempotency before processing
- If duplicate: return original result
- No double processing possible
Edge case 4: April 2025 Outage (Real Incident)
What happened:
- Some PSP banks were calling the "Check Transaction Status" API excessively, without waiting for responses
- Some re-checked transactions from previous days repeatedly
- Usage limits on this API weren't enforced at the switch (NPCI added rate limiters afterwards)
- The switch was flooded: on 12 April 2025 the success rate fell to about 50% for nearly two hours
Lesson learned:
- Rate limit ALL APIs, not just transaction APIs
- Enforce guidelines at NPCI firewall, not just bank side
- Separate critical path APIs from status check APIs
- Circuit breaker for misbehaving banks
Scaling and edge cases
Q1/3 A user's app times out and they tap Pay again, but the first request already succeeded. What prevents a double debit?
Answer & explanation
Answer: B
Idempotency is the guarantee: NPCI looks up the key first and, on a duplicate, returns the stored result. Rate limits and breakers manage load; a single retry sits well within them and would still go through.
Q2/3 SBI, the source of roughly a quarter of transactions, goes down. What is the right behavior for the rest of the system?
Answer & explanation
Answer: C
The money lives in SBI accounts, so no other bank can move it; the right move is a fast, clear failure for SBI and isolation for everyone else. NPCI failover does not help, because the switch is healthy; the outage is at the bank.
Q3/3 What was the core lesson of the April 2025 outage?
Answer & explanation
Answer: D
Banks hammered the unlimited "Check Transaction Status" API and it degraded the whole system. The fix is rate limiting all APIs at NPCI and isolating status checks from transactions, not tuning timeouts or caches.
Phase 6: Monitoring and Operations#
"For a system processing nearly ₹30 trillion a month (September 2026), monitoring isn't optional."
Key Metrics Dashboard#
Real-time health
| Transaction Rate | Success Rate | Latency (p99) |
|---|---|---|
| 7,842 TPS | 99.2% | 1,187 ms |
| (Target: 10K) | (Target: 99.0%) | (Target: 2,000ms) |
Bank health matrix
| Bank | TPS | Success | Latency | Circuit | Issues |
|---|---|---|---|---|---|
| SBI | 2,341 | 98.7% | 423ms | CLOSED | Minor lag |
| HDFC | 1,856 | 99.8% | 287ms | CLOSED | Healthy |
| ICICI | 1,234 | 99.5% | 312ms | CLOSED | Healthy |
| Axis | 987 | 94.2% | 892ms | HALF-OPEN | HIGH LATENCY |
| Kotak | 654 | 99.1% | 345ms | CLOSED | Healthy |
| Yes Bank | 543 | 99.4% | 298ms | CLOSED | Healthy |
| ... |
Alerts (last 1 hour)
- 14:32 — Axis Bank latency > 800ms (CRITICAL)
- 14:28 — SBI error rate 1.3% (Warning)
- 14:15 — Axis Bank circuit half-open (Info)
- 13:45 — Traffic spike +20% (Auto-scaled)
SLOs for UPI#
SLO 1: Transaction success rate
- Target: 99.0% of transactions succeed
- Measurement: Successful / Total (excluding user errors)
- Current: 99.2%
- Exclusions:
- Insufficient balance (user error)
- Wrong PIN (user error)
- Account blocked (compliance)
SLO 2: End-to-end latency
- Target: 99% of transactions complete in < 2 seconds
- Measurement: Time from request received to response sent
- Current: p99 = 1.2 seconds
- Breakdown:
- NPCI processing: < 300ms
- Bank response (each of debit and credit): < 700ms
- Network overhead: < 200ms
SLO 3: Availability
- Target: 99.9% uptime
- Measurement: (Total time - Downtime) / Total time
- Allowed downtime: 8.7 hours/year
SLO 4: Money safety
- Target: 100% of debited amounts credited or reversed
- Measurement: No money stuck beyond T+1 (RBI turnaround time for P2P)
- Current: 99.99% resolved within 24 hours
This is NON-NEGOTIABLE. Error budget = 0
SLOs for money
Q1/3 The success-rate SLO is 99.0%, yet the money-safety SLO has an error budget of zero. Why the difference?
Answer & explanation
Answer: B
A failed payment costs the user a retry. Money that leaves an account and is neither credited nor reversed breaks the one promise a payment system makes, so every such case must be resolved, which is why reconciliation and T+1 reversal exist.
Q2/3 The success-rate SLO excludes insufficient balance and wrong PIN. Why?
Answer & explanation
Answer: C
An SLO should move when the system misbehaves. Counting users' own declined payments would burn the error budget on things engineers can't fix and blur the signal they can.
Q3/3 The bank health matrix shows Axis Bank at 94.2% success, 892 ms latency and a HALF-OPEN circuit. What is the switch doing for Axis right now?
Answer & explanation
Answer: B
HALF-OPEN is the probe state: after the open timeout, a few trial calls go through. Enough consecutive successes close the breaker; any failure sends it back to OPEN.
Final Challenge: The Flash-Sale Merchant#
The interviewer leans back. "Last one. A curveball."
"A huge online sale opens at noon, and one merchant's payment address receives 50,000 payments a minute for an hour. Everything else stays normal. Where does your design get hot, and what do you change?"
- One payee VPA, 50,000 incoming payments a minute, for an hour
- Payers are spread across hundreds of banks; the merchant banks with one
- Every payer expects the usual few-second confirmation
- Payers who time out will tap Pay again
Find the hot spots before you fix anything
Trace a single payment through your design and mark every place where all 50,000 a minute meet the same key, row or queue. Which of those are reads, which are writes, and what would you do for each?
One strong answer
- The payee lookup (a read): every payment resolves the same VPA. That's a hot key, and the easy one: it's already in the switch's VPA cache, and a short-lived local copy on each switch node takes it off the shared cache entirely.
- The merchant's account (the real write hotspot): every credit updates the same balance in the merchant's bank, so the bank serialises them on one row. Fix it by spreading credits: append ledger entries and compute the balance asynchronously, or credit several sub-accounts and roll them up.
- Duplicates: retries come from payers who timed out, so every retry must carry the original transaction ID, and the switch returns the original result instead of debiting twice.
- Guards that must not misfire: per-user rate limits and velocity checks are for payers. Applied to the receiving merchant, they would reject honest customers.
- Notifications: batch or aggregate the merchant's confirmations rather than pushing 50,000 separate alerts a minute to one device.
Final challenge · Quest check
Q1/4 50,000 payments a minute to one merchant is about how many per second?
Answer & explanation
Answer: B
50,000 ÷ 60 ≈ 833 per second. That's a small share of the switch's ~28,000 TPS peak, which is the point: the problem isn't total volume, it's that all of it lands on one key and one account.
Q2/4 Where is the hardest hot spot in the flash-sale scenario?
Answer & explanation
Answer: B
Reads of one key are cheap to cache. Writes to one balance row have to be serialised, so they are the real bottleneck. The switch's own transaction rows are spread across partitions by transaction ID, and the payers' debits are spread over many accounts and banks.
Q3/4 A payer's app times out after they tap Pay, and they tap again. What prevents a double debit?
Answer & explanation
Answer: C
The idempotency key is created once, with the payment, and reused on every retry. A key built from the retry's own timestamp would never match, and an amount-based rule would reject genuine repeat purchases.
Q4/4 Why should a "5 payments per minute" velocity limit apply to payers and not to the receiving merchant?
Answer & explanation
Answer: B
Velocity rules model normal behaviour for a role. Five payments a minute is a lot for one person to send and nothing for a busy merchant to receive, so the same threshold on the payee side would turn a fraud guard into an outage.
Interview Conclusion#
"Impressive depth. A few rapid-fire questions:"
"Why didn't India just adopt an existing system like Visa/Mastercard?"
"Three reasons:
- Cost: Card payments typically cost merchants ~1-3% (merchant discount rate). UPI is nearly free.
- Inclusion: Cards need plastic issuance and POS terminals (and credit cards need credit checks). UPI needs only a phone.
- Control: Critical financial infrastructure shouldn't depend on foreign companies.
The result: UPI enabled the chai vendor to accept digital payments for a ₹10 tea."
"What's the biggest technical achievement of UPI?"
"Interoperability without centralized money holding. NPCI routes transactions but never touches the money. This means:
- No counterparty risk (NPCI can't go bankrupt with your money)
- Banks remain the regulated entities
- Scales horizontally (NPCI is just a switch)
- Any app works with any bank
This architecture is why countries worldwide are studying UPI."
"If you were to improve UPI today, what would you change?"
"Based on the April 2025 outage:
- Stricter rate limiting at NPCI level, not trusting banks to self-regulate
- Better isolation between critical transaction APIs and status check APIs
- More granular circuit breakers — per-API, not just per-bank
- Chaos engineering — regularly test failure scenarios in production"
Summary: Concepts Applied from 10-Week Course#
- Week 1: Data at scale
- Partitioning — VPAs partitioned by bank suffix
- Replication — Multi-DC active-active setup
- Read optimization — VPA caching at NPCI level
- Week 2: Failure-first design
- Timeouts — Strict timeouts for bank calls (30s)
- Circuit breakers — Per-bank failure isolation
- Idempotency — Transaction idempotency keys
- Retries — Smart retry with exponential backoff
- Week 3: Messaging & async
- Transactional outbox — Audit logging
- Dead letter queues — Failed reversal handling
- Event streaming — Transaction events for reconciliation
- Week 4: Caching
- VPA resolution caching
- Bank configuration caching
- Negative caching — Non-existent VPAs
- Week 5: Consistency & coordination
- Distributed transactions — Debit-then-credit with rollback
- State machine — Transaction lifecycle management
- Exactly-once semantics — Idempotency guarantees
- Week 9: Security & compliance
- Multi-factor authentication — Device + PIN
- Encryption — Public-key PIN encryption, HSM-based PIN verification
- Fraud detection — Real-time ML scoring
- Audit trail — Complete transaction logging
- Week 10: Production readiness
- SLOs — Success rate, latency, availability targets
- Observability — Per-bank health dashboards
- Capacity planning — Festival traffic handling
- Incident management — April 2025 outage learnings
Why UPI Matters#
Scale
- ~49% of the world's real-time payment volume (2023)
- More than Visa + Mastercard combined (in India)
- ~800 million transactions DAILY
Inclusion
- Works on ₹3,000 smartphones
- Works on 2G networks
- ₹1 transactions viable (no minimums)
- Brought hundreds of millions of first-time users into digital payments
Cost
- Zero cost to consumers
- Near-zero cost to small merchants
- Saved billions in card network fees
Innovation
- VPA system (email for money)
- Interoperable (any app, any bank)
- Open standard (countries can adopt)
- Built on existing bank infrastructure
Global impact
- 11 countries accepting UPI (August 2026)
- More countries in talks to adopt it
- Model for BIS cross-border payment initiatives
"UPI proved that with the right architecture, a developing nation can leapfrog decades of financial infrastructure and build something the developed world envies."
Self-Assessment Checklist#
After studying this case study, you should be able to:
Architecture:
- Explain the three-layer UPI architecture (Apps → NPCI → Banks)
- Design a VPA resolution system with caching
- Implement atomic transactions with rollback capability
Distributed Systems:
- Handle partial failures in multi-party transactions
- Implement circuit breakers for unreliable dependencies
- Design idempotency for payment systems
Scale:
- Calculate infrastructure needs for billion-transaction systems
- Plan for bursty traffic (festivals, events)
- Implement graceful degradation under load
Security:
- Design multi-factor authentication for payments
- Implement real-time fraud detection
- Understand HSM-based PIN verification
Operations:
- Define meaningful SLOs for payment systems
- Monitor multi-party systems (NPCI + 740 banks)
- Learn from production incidents (April 2025 outage)
Sources#
Statistics and Data:
- Read
NPCI Official UPI Statistics: https://www.npci.org.in/product/upi/product-statistics
- Read
Outlook Money - UPI September 2026 Data: https://www.outlookmoney.com/banking/upi-processes-2407-billion-transactions-worth-rs-2937-lakh-crore-in-september
- Read
Business Today - UPI Calendar 2025 Totals: https://www.businesstoday.in/technology/news/story/upi-ends-2025-on-a-high-with-record-monthly-and-annual-transactions-with-plans-for-future-initiatives-509218-2026-01-02
- Read
Drishti IAS - UPI at 10 Years (banks live, countries): https://www.drishtiias.com/daily-updates/daily-news-analysis/unified-payments-interface-upi-1
- Read
LatestLY / Lok Sabha Reply - UPI Users Onboarded (June 2026): https://www.latestly.com/agency-news/business-news-upi-user-base-reaches-55-49-crore-fy26-transactions-rise-to-24162-crore-worth-rs-314-lakh-crore-7524680.html
- Read
ACI Worldwide - Prime Time for Real-Time 2024: https://www.businesswire.com/news/home/20240429624485/en
- Read
Business Today - Diwali 2025 UPI Peak: https://www.businesstoday.in/amp/personal-finance/news/story/diwali-lights-up-upi-record-rs-102-lakh-crore-transactions-amid-festive-fervour-lower-gst-rates-499190-2025-10-22
- Read
Business Standard - UPI December 2024 Data: https://www.business-standard.com/finance/news/upi-transactions-surge-to-record-16-73-bn-in-dec-value-at-rs-23-25-trn-125010100457_1.html
- Read
Inc42 - Technical Declines Fall to 0.7-0.8% (NPCI CEO, November 2024): https://inc42.com/buzz/technical-glitches-on-upi-have-fallen-below-1/
- Read
Free Press Journal - SBI Top Remitter, 5.2 Billion Transactions (August 2025): https://www.freepressjournal.in/business/upi-transactions-surge-to-90446-crore-daily-average-in-august-sbi-leads-with-52-billion-transactions
- Read
Angel One - HDFC Bank UPI Maintenance Downtime (December 2025): https://www.angelone.in/news/stocks/hdfc-bank-upi-downtime-scheduled-for-december-2025-dates-timings-and-affected-services
- Read
DemandSage UPI Statistics 2025: https://www.demandsage.com/upi-statistics/
- Blog
Meetanshi UPI Statistics: https://meetanshi.com/blog/upi-statistics/
- Read
GrabOn UPI Statistics: https://www.grabon.in/indulge/tech/upi-statistics/
Architecture and Technical Details:
- Wiki
Wikipedia - Unified Payments Interface: https://en.wikipedia.org/wiki/Unified_Payments_Interface
- Docs
ByteByteGo - UPI Architecture: https://bytebytego.com/guides/unified-payments-interface-upi-in-india/
- Read
GeeksforGeeks - Designing UPI System Design: https://www.geeksforgeeks.org/designing-upi-system-design/
- Read
Medium - Deep Dive System Design of UPI: https://medium.com/@avinashkariya05910/deep-dive-system-design-of-upi-unified-payments-interface-eff3b0334b0d
- Read
Brickendon Consulting - UPI Technical Overview: https://www.brickendon.com/insights/unified-payments-interface-upi/
- Read
Dev.to - System Design UPI: https://dev.to/zeeshanali0704/system-design-upi-unified-payment-interface-2ng3
Infrastructure and Security:
- Docs
ITNews Asia - NPCI Data Center Modernization: https://www.itnews.asia/news/indias-npci-modernises-data-centres-using-kyndryls-cloud-services-592061
- Paper
NPCI Smart Data Center Press Release: https://www.npci.org.in/PDF/npci/press-releases/2020/NPCI_Press_Release-NPCI_to_launch_Smart_Data_Center_in_Hyderabad.pdf
- Read
The420.in - UPI Security Infrastructure: https://the420.in/upi-digital-payments-infrastructure-security-npci-cpt-analysis/
- Blog
Blog - UPI Security Architecture Deep Dive: https://blog.akshanshjaiswal.com/the-upi-architecture-a-security-look
Settlement and Operations:
- Blog
BillCut - Settlement Latency Benchmarks: https://www.billcut.com/blogs/settlement-latency-benchmarks-whos-fastest/
- Blog
Razorpay - UPI Payout Processing: https://razorpay.com/blog/business-banking/payout-processing-imps-upi-transactions-deemed-success-npci/
- Read
Inc42 - NPCI Real-Time Fix for UPI Failures: https://inc42.com/buzz/ncpi-working-on-real-time-fix-for-upi-transaction-failures/
- Paper
BIS Papers - Faster Digital Payments India: https://www.bis.org/publ/bppdf/bispap152_e_rh.pdf
Outage Analysis (April 2025):
- Wiki
Wikipedia - UPI Outage Details: https://en.wikipedia.org/wiki/Unified_Payments_Interface
- Docs
Outlook Money - NPCI on the April 12 Outage: https://www.outlookmoney.com/banking/upi-outage-attributed-to-surge-in-api-calls-from-banks-says-npci
- Docs
Inc42 - NPCI Limits Check Transaction API: https://inc42.com/?p=511758
Further Reading#
Official Documentation:
- Read
NPCI Official Website: https://www.npci.org.in/
- Read
NPCI UPI Product Page: https://www.npci.org.in/what-we-do/upi/product-overview
- Read
RBI Payment Systems: https://www.rbi.org.in/Scripts/PaymentSystems_UM.aspx
- Paper
UPI Procedural Guidelines: https://yashada.org/yashada_2019/pdfs/e_library_cit/edpri_UPI_Procedural_Guidelines.pdf
Engineering Blogs and Technical Deep Dives:
- Blog
Razorpay Engineering Blog: https://razorpay.com/blog/ (Multiple articles on UPI integration)
- Blog
Paytm Engineering: https://paytm.com/blog/ (UPI transaction insights)
- Blog
ByteByteGo Newsletter: https://blog.bytebytego.com/ (System design breakdowns)
- Blog
LinkedIn Engineering Posts: Search "UPI Architecture" for practitioner insights
Research Papers and Reports:
- Paper
BIS Papers No. 152: Faster Digital Payments - Global and Regional Perspectives (India Chapter)
- Paper
NPCI White Papers: Available on NPCI website
- Read
RBI Annual Reports: Digital payments statistics and trends
News and Industry Analysis:
- Read
Medianama: https://www.medianama.com/ (Digital payments coverage)
- Read
Economic Times Tech: https://economictimes.indiatimes.com/tech (Fintech news)
- Read
Inc42: https://inc42.com/ (Startup and fintech coverage)
- Read
LiveMint: https://www.livemint.com/ (Financial news)
Video Resources:
- Talk
NPCI YouTube Channel: Official explainers and announcements
- Talk
System Design Interview Videos: Search "UPI System Design" on YouTube
Books:
- Book
"Designing Data-Intensive Applications" by Martin Kleppmann - Foundational concepts
- Book
"System Design Interview" by Alex Xu - Interview preparation with similar patterns
Related Systems to Study:
- Read
PIX (Brazil): Similar instant payment system
- Read
FedNow (USA): US real-time payment system
- Read
SEPA Instant (Europe): European instant payments
"A payment system that serves a billion people, handles trillions in transactions, and costs nothing to use. This is what engineering at scale looks like."
this lesson in shorts · 3
play them all →One idea from this lesson in about a minute each, made for your phone.
- 0/7stages cleared
Follow-up drills
0 of 10 practisedYou presented India's UPI. Now the interviewer pushes back. Answer each follow-up out loud or in the box, then reveal a strong answer.
“You said ~28,000 to 46,000 TPS at peak. Walk me from 800 million a day to that number. Is that what you size the switch for?”
Pushes on · sizing on transactions instead of the work each one causes
Go deeper · reading trail
all reading →Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.
Five Years of Kafka at Razorpay's UPI Switch
Kshitij Nawandar · 2026- What
- The story of the messaging layer inside Razorpay's UPI Switch, which talks to NPCI for most of its UPI volume.
- Key idea
- UPI is asynchronous, so a message bus sits at the heart of the switch; SQS fan-out gave way to Kafka, then topics were consolidated when partitions exploded.
- Why here
- What 'bank integration at scale' looks like inside a payment service provider: every UPI callback is an event fanned out to NPCI updates, merchant callbacks and the warehouse.
Demystifying TStore: The Backbone of Billions of Transactions at PhonePe
Arnab Bir, Tushar Naik · 2024- What
- PhonePe's transaction store behind the green success screen and the payment history feed.
- Key idea
- Every service in a payment flow writes to TStore, which keeps one consolidated record per transaction and ingests around 45,000 writes a second at peak.
- Why here
- The UPI app side of this quest: a pending payment must show one consistent status even while the bank and NPCI are still answering.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.


