?

Side quest 01Real-time payments

India's UPI.

The World's Largest Real-Time Payment System

INTERMEDIATE40 MIN READ6 PHASES+120 XP

Mission brief

Design a real-time inter-bank payment system

Quest log0/7 stages · 0/5 checkpoints · 0/7 quick checks
transactions a day (September 2026)
~800M
participating banks
740+
end to end, p99
< 2 s
peak transactions per second to design for
~28-46K

Company names and logos are trademarks of their owners, shown only to identify each company. This course isn't affiliated with or endorsed by any of them.

watch it first · newThe whole design in a 16-minute animated lecture4 quiz checkpoints · ask the AI about any moment · captions, chapters and speed
Optional · timed run · 6 phases

Want it harder? Hide the model answer and work each phase in a scratchpad, then reveal and score yourself. Or simply read on: the checkpoints below work either way.

Intro · 2 min

A Revolution That Changed a Nation

In 2016, India launched an experiment. Could a country of 1.4 billion people, many unbanked, leapfrog decades of payment infrastructure and go directly to real-time digital payments?

Ten years later, the answer is extraordinary.

UPI payment flow · app → PSP → NPCI switch → payer and payee banks, while the money settles bank to bank
LIVEUPI payment flow · app → PSP → NPCI switch → payer and payee banks, while the money settles bank to bank
The numbers that define UPI (as of 2026)
  • ~800 Million — daily transactions (802M average, September 2026)
  • 24+ Billion — monthly transactions (24.07B, September 2026)
  • 228 Billion — annual transactions (calendar 2025)
  • $3.5 Trillion (₹300 Lakh Crore) — annual value (calendar 2025)
  • 740+ — participating banks (741, July 2026)
  • 550+ Million — users onboarded (June 2026)
  • A few seconds — typical end-to-end payment time
  • ~99% — technical success rate (technical declines at 0.7-0.8%, NPCI's CEO, November 2024)
  • ~49% of the world's real-time payment volume — global share (2023 data, ACI Worldwide)
  • 11 (Singapore, UAE, France, Nepal, Bhutan, Sri Lanka, Mauritius, Qatar...) — countries accepting UPI (August 2026)
For context
  • UPI processes MORE transactions than Visa and Mastercard COMBINED in India
  • A tea vendor in rural India uses the same system as a Fortune 500 company
  • Transactions as small as ₹1 (about 1 US cent) work with the same reliability as ₹1 Crore
  • The NPCI switch runs 24/7/365; individual banks still announce short maintenance windows (HDFC Bank did several in 2025)

This is the system we'll design today.

ExplainerThe whole quest: India's UPI

UPI moves about 800 million payments a day between 740+ banks in under 2 seconds: ~9,300 TPS on average, 28,000 to 46,000 at peak. Apps (PSPs) talk to the NPCI switch, which resolves the VPA, debits the payer's bank and credits the payee's while money settles bank to bank through the RBI. The deep dives cover cached VPA resolution, a state machine with DEEMED_SUCCESS and auto-reversal by T+1, per-bank circuit breakers and layered security, then a 5x Diwali, data-centre failover, idempotent retries and the April 2025 status-check outage.

The Interview Begins#

You're interviewing at a fintech company. The principal architect draws on the whiteboard:

Interviewer

"India's UPI is considered one of the greatest achievements in financial technology. Countries around the world are trying to replicate it. Today, I want you to design a system like UPI from scratch."

Design a Real-Time Inter-Bank Payment System

Build the infrastructure that enables instant money transfers between ANY two bank accounts in a country using just a phone.

Requirements
  • Support 700+ banks with different legacy systems
  • Handle 800+ million transactions per day
  • Complete each transaction in < 2 seconds end-to-end
  • 99.9% availability (< 9 hours downtime/year)
  • Zero tolerance for money loss (atomic transactions)
  • Work on basic smartphones with 2G/3G connectivity
  • Support both P2P (person-to-person) and P2M (person-to-merchant)
Interviewer

"This is arguably one of the hardest system design problems — you're building financial infrastructure for a nation. Take your time."

Phase 1 · Stage 1 of 7 · 2 min

Phase 1: Requirements Clarification#

You

"Before I start, let me understand the constraints better."

Your Questions#

Transcript
You

"First, what's the relationship between banks and this central system? Do banks connect directly, or through intermediaries?"

Interviewer

"Banks connect through the central switch — NPCI in India's case. They don't talk to each other directly. The central system orchestrates everything."

You

"What about the mobile apps? Can any app connect to the system?"

Interviewer

"Apps must be approved and must partner with a bank. We call these Payment Service Providers or PSPs. PhonePe started with Yes Bank (hence @ybl), Google Pay works with multiple banks. The app itself doesn't hold money — it's just an interface."

You

"How do users identify each other? Bank account numbers are long and error-prone."

Interviewer

"Great observation. UPI solved this with Virtual Payment Addresses — like an email for money. username@bankname. The system maps this to actual bank accounts."

You

"What happens if a transaction fails mid-way? Say money is debited but not credited?"

Interviewer

"This is critical. The system MUST be atomic. Either the full transaction succeeds, or it's completely rolled back. Users cannot lose money due to technical failures."

You

"What's the peak traffic pattern? Is it bursty?"

Interviewer

"Very bursty. Evenings run well above the daily average, and festival days like Diwali set new records. Plan for peaks of 3-5x the average so the system handles them gracefully."

Requirements Summary#

Functional Requirements:

Functional requirements
1. User management
  • Register users via mobile number + bank account
  • Create and manage Virtual Payment Addresses (VPAs)
  • Link multiple bank accounts to one app
  • Two-factor authentication (device binding + PIN)
2. Payment operations
  • Push payments (I send money to you)
  • Pull payments / Collect requests (I request money from you)
  • QR code payments (scan and pay)
  • Recurring payments (autopay/mandates)
3. Transaction processing
  • Real-time debit from sender's bank
  • Real-time credit to receiver's bank
  • Transaction status tracking
  • Refund/reversal handling
4. Bank integration
  • Standardized APIs for all banks
  • VPA to account resolution
  • Balance inquiry (with consent)
  • Account validation
5. Settlement
  • Net settlement between banks (periodic)
  • Reconciliation and dispute handling
  • Audit trail for compliance

Non-Functional Requirements:

Non-functional requirements
Scale
  • 800+ million transactions/day
  • 24+ billion transactions/month
  • 550+ million users
  • 740+ participating banks
Latency
  • End-to-end: < 2 seconds (p99)
  • NPCI switch processing: < 300ms
  • Bank response time: < 700ms per call (debit and credit)
Availability
  • 99.9% uptime (8.7 hours downtime/year max)
  • 24/7/365 operation
  • No scheduled maintenance windows
Consistency
  • ACID transactions (money can't be lost)
  • Exactly-once semantics
  • Atomic debit-credit operations
Security
  • End-to-end encryption
  • Device binding
  • Multi-factor authentication
  • Fraud detection in real-time

Phase 2 · Stage 2 of 7 · 2 min

Phase 2: Back of the Envelope Estimation#

You

"Let me work through the numbers..."

Your turn · checkpoint 1not answered

Before the numbers: what does 800 million a day mean per second?

Payments run at about 800 million a day, and you plan for evening and festival peaks of 3 to 5 times the average. Work out the average and the peak transactions per second. Then: every payment also needs a payee lookup, two bank calls, audit writes and two notifications. Roughly how many backend operations per second is the peak?

Think it through, or jot it down. Then compare.

One strong answer

  • Average: 800,000,000 ÷ 86,400 s ≈ 9,300 TPS.
  • Peak: 3 to 5 times that, so ~28,000 to 46,000 TPS.
  • Fan-out: each payment is about six operations (a lookup, a debit call, a credit call, one or two audit writes, two pushes), so the peak is ~170,000+ operations per second. That fan-out, not the raw payment count, is what you size the switch, the queues and the bank links for.

Traffic Calculations#

WorksheetTransactions per second
  • Daily transactions: 800,000,000
  • Seconds per day: 86,400
  • Average TPS: ~9,300 TPS
  • Peak multiplier: 3-5x (design headroom for evenings, festivals)
  • Peak TPS: ~28,000-46,000 TPS
  • Per transaction, multiple operations:
    • VPA resolution: 1 lookup
    • Sender bank call: 1 API call
    • Receiver bank call: 1 API call
    • Audit logging: 1-2 writes
    • Notifications: 2 pushes
  • Effective operations/second: ~170,000+ at peak

Data Volume#

WorksheetStorage requirements
  • Per transaction record: ~500 bytes
    • Transaction ID: 36 bytes (UUID)
    • Sender VPA: 50 bytes
    • Receiver VPA: 50 bytes
    • Amount: 8 bytes
    • Timestamps: 16 bytes
    • Status: 4 bytes
    • Bank references: 100 bytes
    • Metadata: 200 bytes
  • Daily storage: ~900 GB/day
    • Transactions: 800M × 500B = 400 GB/day
    • Audit logs: ~500 GB/day
  • Annual storage: ~330 TB/year
  • 7-year retention: ~2.3 PB

Infrastructure Estimates#

Compute requirements

At ~28,000 TPS peak, assuming each server handles 1,000 TPS:

  • API servers: 28+ servers (with redundancy: 56+)
  • Database: Clustered, sharded
  • Cache: Distributed Redis cluster
  • Message queues: High-throughput Kafka cluster
Network
  • Connections to 740+ banks
  • Each bank: dedicated secure link
  • Geographic distribution: Multiple data centers
  • Bandwidth: Several Gbps
Quick check3 questions · +5 XP each

Requirements and estimation

  1. Q1/3 With 800 million transactions a day and a 3-5x peak multiplier, what peak transaction rate should the switch be sized for?

    Answer & explanation

    Answer: B

    800M / 86,400 s is about 9,300 TPS on average; 3-5x gives roughly 28,000-46,000 TPS at peak. The ~170,000 figure is effective operations per second, not transactions, so sizing transaction capacity on it overshoots.

  2. Q2/3 Why do ~28,000-46,000 peak transactions per second turn into ~170,000+ operations per second?

    Answer & explanation

    Answer: C

    One payment is not one operation: it needs a VPA resolution, a debit call, a credit call, 1-2 audit writes and 2 pushes. Retries happen, but they are not what multiplies the load in the estimate.

  3. Q3/3 Storage comes to ~900 GB/day (transactions plus audit logs). What does 7-year retention need, roughly?

    Answer & explanation

    Answer: D

    900 GB x 365 is roughly 330 TB a year, and 7 years of that is about 2.3 PB. 200 TB undercounts by forgetting the multi-year retention; 20 PB is an order of magnitude too high.

Phase 3 · Stage 3 of 7 · 3 min

Phase 3: High-Level Architecture#

You

"Let me draw how UPI actually works. It's a beautiful layered architecture."

Your turn · checkpoint 2not answered

Sketch it first: who holds the money?

Hundreds of banks, dozens of apps, one central switch. Sketch the layers. Then decide: while a payment is in flight, should the central switch hold the money itself?

Think it through, or jot it down. Then compare.

One strong answer

Three layers:

  • Apps (PSPs): each app partners with a bank, onboards users and creates their payment addresses. The app never holds money.
  • The central switch (NPCI): routes every request between banks and runs the transaction state machine.
  • Banks: hold the accounts and post the debit and the credit.

The switch must not hold money. Information flows app → PSP → switch → banks, while money moves bank to bank: customer accounts are debited and credited in real time, and the interbank obligations are netted and settled in batches through the banks' accounts at the central bank. So the switch carries no counterparty risk, banks stay the regulated entities, and the switch can scale out like any stateless router plus a transaction log.

The Three-Layer Cake#

DiagramUPI architecture — the three-layer cake
%% title: UPI architecture — the three-layer cake
flowchart TD
  subgraph L1["Layer 1 · User interface"]
    PP@{ icon: "sd:phone", label: "PhonePe (App)" }
    GP@{ icon: "sd:phone", label: "Google Pay (App)" }
    PT@{ icon: "sd:phone", label: "Paytm (App)" }
    BH@{ icon: "sd:phone", label: "BHIM (App)" }
    PSP["PSP (Payment Service Providers)<br/>User onboarding · VPA creation · UI/UX"]:::edge
  end
  PP & GP & PT & BH --> PSP
  subgraph L2["Layer 2 · NPCI switch"]
    subgraph NPCI["NPCI UPI platform"]
      VPA["VPA Mapper"]:::service
      TR["Transaction Router"]:::hot
      FD["Fraud Detection"]:::service
      SE["Settlement Engine"]:::service
      AT[("Audit Trail")]:::store
      DR["Dispute Resolution"]:::service
      direction TB
      VPA ~~~ SE
      TR ~~~ AT
      FD ~~~ DR
    end
  end
  PSP ==>|"information"| NPCI
  subgraph L3["Layer 3 · Banking layer"]
    direction TB
    SBI@{ icon: "sd:bank", label: "SBI Bank" }
    HDFC@{ icon: "sd:bank", label: "HDFC Bank" }
    ICICI@{ icon: "sd:bank", label: "ICICI Bank" }
    AXIS@{ icon: "sd:bank", label: "Axis Bank" }
    MORE@{ icon: "sd:bank", label: "740+ Banks" }
    IMPS[["IMPS rails + RBI net settlement<br/>how money moves"]]:::queue
  end
  NPCI ==>|"information"| L3
  SBI & HDFC & ICICI & AXIS & MORE -->|"money"| IMPS
  class PP,GP,PT,BH client
  class SBI,HDFC,ICICI,AXIS,MORE external
  • PSP (Payment Service Providers): apps must partner with a bank (PSP Bank) to access UPI — PhonePe → Yes Bank, Google Pay → multiple banks. The PSP handles user onboarding, VPA creation and UI/UX.
  • NPCI UPI platform: the brain of UPI — routes transactions between banks. NPCI NEVER holds money — only routes information.
  • IMPS (Immediate Payment Service) and settlement: UPI is built on the IMPS rails — customer accounts are debited and credited in real time, while the interbank obligations are netted by NPCI and settled in batches (several cycles a day) through the banks' accounts at the RBI.

KEY INSIGHT:

  • Money flows: Bank → Bank (real-time account postings; net interbank settlement via RBI)
  • Information flows: App → PSP → NPCI → Banks
  • NPCI is the orchestrator, not a money holder

Transaction Flow#

You

"Let me trace a ₹500 payment from Alice to Bob..."

DiagramTransaction flow — Alice pays Bob ₹500
%% title: Transaction flow — Alice pays Bob ₹500
sequenceDiagram
  autonumber
  participant A as Alice's phone (PhonePe app)
  participant N as NPCI switch
  box Banks
    participant H as HDFC (Alice's bank)
    participant X as Axis (Bob's bank)
  end
  A->>N: INITIATE · "Pay ₹500 to bob@okaxis"
  Note over A,N: + Alice's VPA<br/>+ Encrypted PIN
  N->>X: RESOLVE VPA · "Who is bob@okaxis?"
  X-->>N: VPA RESPONSE · "Bob's account at Axis Bank: XXXX1234"
  N->>H: DEBIT REQUEST · "Debit ₹500 from Alice at HDFC"
  H-->>N: DEBIT RESPONSE · "Debited. Ref: ABC123"
  N->>X: CREDIT REQUEST · "Credit ₹500 to Bob at Axis"
  X-->>N: CREDIT RESPONSE · "Credited. Ref: XYZ789"
  N-->>A: SUCCESS · "Payment complete! Ref: TXN123456"
  Note over A,X: TOTAL TIME: < 2 seconds

PARALLEL ACTIONS:

  • Audit log written at each step
  • Fraud check runs during step ①
  • Push notifications sent to both Alice and Bob
  • Settlement record created for bank reconciliation
Quick check2 questions · +5 XP each

High-level architecture

  1. Q1/2 Alice pays a merchant through PhonePe. Where does her money actually move?

    Answer & explanation

    Answer: A

    Money flows bank to bank (real-time postings, net interbank settlement via RBI); information flows App -> PSP -> NPCI -> Banks. The PSP app is only an interface and NPCI is an orchestrator that never holds money, so neither is a hop for the funds.

  2. Q2/2 In the Alice-to-Bob flow, why does NPCI resolve bob@okaxis before sending the debit to HDFC?

    Answer & explanation

    Answer: C

    The credit leg needs the real account behind the VPA, and resolving first means an invalid VPA fails cleanly with no debit to undo. Fraud checks run in parallel at initiation; they are not what VPA resolution is for.

Phase 4 · Stage 4 of 7 · 19 min

Phase 4: Deep Dives#

Deep Dive 1: Virtual Payment Address (VPA) Resolution#

Week 1 concepts: Partitioning, lookup optimization. Week 4 concepts: Caching.

You

"VPA resolution is called for EVERY transaction. With 800 million daily transactions, this lookup must be blazing fast."

The Challenge:

VPA resolution challenge
Hundreds of millions of VPAs like
  • alice@okhdfcbank
  • bob@okaxis
  • merchant@paytm
  • 9876543210@ybl
  • ...
Each VPA maps to
  • Bank code
  • Account number (encrypted)
  • Account holder name
  • Status (active/blocked)
  • Metadata
Requirements
  • Lookup latency: < 10ms
  • 100% accuracy (wrong mapping = money to wrong person!)
  • Real-time updates (user changes bank)
  • Handle 50,000+ lookups/second at peak

The Solution:

DiagramVPA mapper architecture — VPA resolution flow
%% title: VPA mapper architecture — VPA resolution flow
flowchart TD
  IN(["VPA: bob@okaxis"]):::client
  P["PARSE HANDLE<br/>handle='bob' · suffix='okaxis'"]:::service
  S["SUFFIX LOOKUP<br/>'okaxis' → Axis Bank Code"]:::core
  BR@{ icon: "sd:database", label: "BANK REGISTRY<br/>okhdfcbank → HDFC · okaxis → Axis<br/>oksbi → SBI · ybl → Yes Bank" }
  C{"CACHE CHECK (Redis)<br/>Cache hit?"}:::cache
  Q["QUERY BANK (Real-time)"]:::service
  AX@{ icon: "sd:bank", label: "Axis Bank<br/>VPA Database" }
  RET(["RETURN ACCOUNT DETAILS<br/>Bank: Axis · Account: ***234<br/>Name: Bob Kumar"]):::ok
  IN --> P --> S
  S --> BR
  S --> C
  C ==>|"Yes"| RET
  C -->|"No"| Q
  Q --> AX
  Q -->|"Update cache"| RET
  class BR store
  class AX external

KEY DESIGN DECISIONS (a likely design; NPCI doesn't publish its switch internals):

  1. DISTRIBUTED OWNERSHIP — Each bank owns their VPA namespace (suffix). NPCI doesn't store all VPAs — banks do. This scales naturally as banks handle their own data.
  2. CACHING STRATEGY — Hot VPAs (frequently used) cached at NPCI. TTL: 15-30 minutes. Invalidation: Banks push updates for VPA changes.
  3. HANDLE UNIQUENESS — handle@suffix is globally unique. Banks ensure uniqueness within their namespace. Cross-bank uniqueness handled by suffix differentiation.
pythonAccountDetails · 179 lines
# vpa/resolver.py

"""
VPA Resolution Service

Maps Virtual Payment Addresses to actual bank accounts.
This is the most critical lookup in the entire system.
"""

from dataclasses import dataclass
from typing import Optional, Tuple
from datetime import datetime, timedelta
import hashlib


@dataclass
class AccountDetails:
    """Resolved account information."""
    bank_code: str
    account_number_masked: str  # Only last 4 digits visible
    account_holder_name: str
    ifsc_code: str
    is_active: bool
    verified_at: datetime


@dataclass
class VPAResolutionResult:
    """Result of VPA resolution."""
    success: bool
    account: Optional[AccountDetails] = None
    error_code: Optional[str] = None
    resolution_time_ms: float = 0
    cache_hit: bool = False


class VPAResolver:
    """
    Resolves VPAs to bank account details.
    
    Design principles:
    - Cache aggressively (VPAs don't change often)
    - Fail fast on invalid formats
    - Banks are the source of truth
    """
    
    def __init__(
        self,
        cache,           # Redis cluster
        bank_registry,   # Bank code → Bank API mapping
        bank_gateway,    # Gateway to call bank APIs
        metrics
    ):
        self.cache = cache
        self.registry = bank_registry
        self.gateway = bank_gateway
        self.metrics = metrics
        
        # Cache settings
        self.cache_ttl = timedelta(minutes=30)
        self.negative_cache_ttl = timedelta(minutes=5)
    
    async def resolve(self, vpa: str) -> VPAResolutionResult:
        """
        Resolve a VPA to account details.
        
        VPA format: handle@suffix
        Example: alice@okhdfcbank, 9876543210@ybl
        """
        start_time = datetime.utcnow()
        
        # Step 1: Parse and validate VPA format
        parsed = self._parse_vpa(vpa)
        if not parsed:
            return VPAResolutionResult(
                success=False,
                error_code="INVALID_VPA_FORMAT"
            )
        
        handle, suffix = parsed
        
        # Step 2: Get bank code from suffix
        bank_code = self.registry.get_bank_for_suffix(suffix)
        if not bank_code:
            return VPAResolutionResult(
                success=False,
                error_code="UNKNOWN_VPA_SUFFIX"
            )
        
        # Step 3: Check cache
        cache_key = f"vpa:{vpa.lower()}"
        cached = await self.cache.get(cache_key)
        
        if cached:
            if cached == "NOT_FOUND":
                return VPAResolutionResult(
                    success=False,
                    error_code="VPA_NOT_FOUND",
                    cache_hit=True
                )
            
            account = AccountDetails(**cached)
            return VPAResolutionResult(
                success=True,
                account=account,
                resolution_time_ms=self._elapsed_ms(start_time),
                cache_hit=True
            )
        
        # Step 4: Query the bank
        try:
            account = await self.gateway.resolve_vpa(
                bank_code=bank_code,
                handle=handle,
                suffix=suffix
            )
            
            if account:
                # Cache the result
                await self.cache.set(
                    cache_key,
                    account.__dict__,
                    ttl=self.cache_ttl
                )
                
                return VPAResolutionResult(
                    success=True,
                    account=account,
                    resolution_time_ms=self._elapsed_ms(start_time),
                    cache_hit=False
                )
            else:
                # Cache negative result (VPA doesn't exist)
                await self.cache.set(
                    cache_key,
                    "NOT_FOUND",
                    ttl=self.negative_cache_ttl
                )
                
                return VPAResolutionResult(
                    success=False,
                    error_code="VPA_NOT_FOUND",
                    resolution_time_ms=self._elapsed_ms(start_time)
                )
                
        except BankTimeoutError:
            return VPAResolutionResult(
                success=False,
                error_code="BANK_TIMEOUT"
            )
        except BankUnavailableError:
            return VPAResolutionResult(
                success=False,
                error_code="BANK_UNAVAILABLE"
            )
    
    def _parse_vpa(self, vpa: str) -> Optional[Tuple[str, str]]:
        """Parse VPA into handle and suffix."""
        if not vpa or '@' not in vpa:
            return None
        
        parts = vpa.lower().strip().split('@')
        if len(parts) != 2:
            return None
        
        handle, suffix = parts
        
        # Validate handle (alphanumeric, 3-50 chars)
        if not handle or len(handle) < 3 or len(handle) > 50:
            return None
        
        # Validate suffix (registered bank suffix)
        if not suffix or len(suffix) < 2 or len(suffix) > 20:
            return None
        
        return handle, suffix
    
    def _elapsed_ms(self, start: datetime) -> float:
        return (datetime.utcnow() - start).total_seconds() * 1000

Deep Dive 2: Atomic Transactions — The Heart of Trust#

Week 2 concepts: Idempotency, failure handling. Week 5 concepts: Distributed transactions, Saga pattern.

You

"The most critical requirement: money cannot disappear. If I debit Alice but fail to credit Bob, Alice must get her money back. Always."

Your turn · checkpoint 3not answered

The credit call just timed out

Alice's bank has debited ₹500. The call to credit Bob's bank timed out, so you don't know whether Bob was credited. What do you do right now, and how do you guarantee the money is never lost and never moved twice?

Think it through, or jot it down. Then compare.

One strong answer

  • Don't guess. Re-sending the debit risks a double debit, and reversing at once is wrong if the credit actually landed.
  • Record the uncertainty: the transaction moves to an in-between state (UPI calls it deemed success) and goes into the settlement file.
  • Resolve it with the same ID: status checks and retries carry the original transaction ID, so every bank call is idempotent.
  • Reconcile: the banks compare the settlement file with their actual credits. If the credit happened, the payment completes. If it didn't, the debit is reversed automatically, by T+1 at the latest under the RBI's turnaround rules.

The guarantee rests on one transaction ID end to end, an explicit state machine, and reconciliation as the backstop.

The Challenge:

The atomicity challenge
Happy path
  1. Debit Alice (HDFC): ₹500 — ok
  2. Credit Bob (Axis): ₹500 — ok

→ Success!

Failure scenarios

Scenario A: Credit fails after debit

  1. Debit Alice: ₹500 — ok (money left Alice's account)
  2. Credit Bob: TIMEOUT — no (did it go through or not?)

→ UNCERTAINTY! Alice lost ₹500?

Scenario B: Network partition

  1. Debit Alice: ok
  2. Credit Bob: Request sent...
  3. Network dies
  4. We don't know the outcome!

→ UNCERTAINTY!

Scenario C: Duplicate request

  1. User clicks "Pay" twice quickly
  2. Two debit requests sent

→ DOUBLE DEBIT! Alice loses ₹1000?

These scenarios CANNOT happen in a payment system.

The Solution:

DiagramA likely transaction state machine (deemed success is UPI's real term; the other states are this design's)
%% title: A likely transaction state machine (deemed success is UPI's real term; the other states are this design's)
stateDiagram-v2
  [*] --> CREATED
  CREATED --> PENDING: Validation passed
  PENDING --> DEBIT_INITIATED: Send to remitter bank
  DEBIT_INITIATED --> FAILED
  DEBIT_INITIATED --> DEBITED
  DEBITED --> CREDIT_INITIATED: Send to beneficiary bank
  CREDIT_INITIATED --> COMPLETED
  CREDIT_INITIATED --> REVERSED
  CREDIT_INITIATED --> DEEMED_SUCCESS: Credit timeout/failure
  DEEMED_SUCCESS --> REVERSED: Auto-reverse by T+1
  DEBIT_INITIATED: DEBIT INITIATED
  CREDIT_INITIATED: CREDIT INITIATED
  FAILED: FAILED · no debit happened
  DEBITED: DEBITED · money left sender
  COMPLETED: COMPLETED · money received
  REVERSED: REVERSED · money back to sender
  DEEMED_SUCCESS: DEEMED SUCCESS · uncertain state
  class CREATED,PENDING,DEBIT_INITIATED,DEBITED,CREDIT_INITIATED core
  class COMPLETED ok
  class FAILED,REVERSED bad
  class DEEMED_SUCCESS queue

DEEMED SUCCESS: Bank didn't respond in time. Settlement happens; if credit actually failed, it must be auto-reversed by T+1.

ExplainerUPI push payment: debit, credit, deemed success

Alice pays ₹500 to bob@okaxis: NPCI resolves the VPA at Axis, debits Alice at HDFC (ref ABC123), credits Bob at Axis (ref XYZ789) and confirms in under 2 seconds, routing only messages while the money moves bank to bank. If the credit call times out, NPCI marks the payment DEEMED_SUCCESS and puts it in the settlement file; when reconciliation finds no credit, the debit is auto-reversed by T+1 with the same transaction ID.

pythonTransactionState · 239 lines
# transaction/processor.py

"""
Transaction Processing with Atomic Guarantees.

This is the most critical code in the entire system.
Money cannot be lost under any circumstances.
"""

from dataclasses import dataclass
from typing import Optional
from enum import Enum
from datetime import datetime, timedelta
import uuid


class TransactionState(Enum):
    CREATED = "created"
    PENDING = "pending"
    DEBIT_INITIATED = "debit_initiated"
    DEBITED = "debited"
    CREDIT_INITIATED = "credit_initiated"
    COMPLETED = "completed"
    FAILED = "failed"
    DEEMED_SUCCESS = "deemed_success"
    REVERSED = "reversed"


@dataclass
class Transaction:
    """A UPI transaction record."""
    txn_id: str
    sender_vpa: str
    receiver_vpa: str
    amount: int  # In paise (smallest unit)
    
    state: TransactionState
    created_at: datetime
    updated_at: datetime
    
    # Bank references
    sender_bank_ref: Optional[str] = None
    receiver_bank_ref: Optional[str] = None
    
    # For idempotency
    idempotency_key: Optional[str] = None
    
    # Reversal tracking
    reversal_initiated: bool = False
    reversal_completed: bool = False


class TransactionProcessor:
    """
    Processes UPI transactions with atomic guarantees.
    
    Key principles:
    1. IDEMPOTENCY: Same request = same result (no double-debit)
    2. ATOMICITY: Either complete success or complete rollback
    3. DURABILITY: State persisted before any bank call
    4. RECOVERABILITY: Can resume from any failure point
    """
    
    def __init__(
        self,
        db,              # Transaction database
        bank_gateway,    # Bank API gateway
        reversal_queue,  # Queue for failed transactions
        audit_log
    ):
        self.db = db
        self.gateway = bank_gateway
        self.reversal_queue = reversal_queue
        self.audit = audit_log
        
        # Timeouts
        self.debit_timeout = timedelta(seconds=30)
        self.credit_timeout = timedelta(seconds=30)
    
    async def process(
        self,
        sender_vpa: str,
        receiver_vpa: str,
        amount: int,
        idempotency_key: str
    ) -> Transaction:
        """
        Process a payment transaction.
        
        CRITICAL: This method must be idempotent.
        Same idempotency_key = same result, always.
        """
        
        # STEP 0: Check idempotency
        existing = await self.db.get_by_idempotency_key(idempotency_key)
        if existing:
            # Return existing result (no reprocessing)
            await self.audit.log("IDEMPOTENT_RETURN", existing.txn_id)
            return existing
        
        # STEP 1: Create transaction record FIRST
        txn = Transaction(
            txn_id=str(uuid.uuid4()),
            sender_vpa=sender_vpa,
            receiver_vpa=receiver_vpa,
            amount=amount,
            state=TransactionState.CREATED,
            created_at=datetime.utcnow(),
            updated_at=datetime.utcnow(),
            idempotency_key=idempotency_key
        )
        
        # Persist BEFORE any bank call
        await self.db.save(txn)
        await self.audit.log("TXN_CREATED", txn.txn_id)
        
        try:
            # STEP 2: Initiate debit
            txn.state = TransactionState.DEBIT_INITIATED
            txn.updated_at = datetime.utcnow()
            await self.db.save(txn)
            
            debit_result = await self.gateway.debit(
                vpa=sender_vpa,
                amount=amount,
                txn_ref=txn.txn_id,
                timeout=self.debit_timeout
            )
            
            if not debit_result.success:
                # Debit failed cleanly — no money moved
                txn.state = TransactionState.FAILED
                txn.updated_at = datetime.utcnow()
                await self.db.save(txn)
                await self.audit.log("DEBIT_FAILED", txn.txn_id, 
                                    debit_result.error)
                return txn
            
            # STEP 3: Debit succeeded — record it
            txn.state = TransactionState.DEBITED
            txn.sender_bank_ref = debit_result.bank_reference
            txn.updated_at = datetime.utcnow()
            await self.db.save(txn)
            await self.audit.log("DEBIT_SUCCESS", txn.txn_id)
            
            # STEP 4: Initiate credit
            # CRITICAL: From this point, we MUST either complete or reverse
            txn.state = TransactionState.CREDIT_INITIATED
            txn.updated_at = datetime.utcnow()
            await self.db.save(txn)
            
            credit_result = await self.gateway.credit(
                vpa=receiver_vpa,
                amount=amount,
                txn_ref=txn.txn_id,
                timeout=self.credit_timeout
            )
            
            if credit_result.success:
                # SUCCESS! Transaction complete
                txn.state = TransactionState.COMPLETED
                txn.receiver_bank_ref = credit_result.bank_reference
                txn.updated_at = datetime.utcnow()
                await self.db.save(txn)
                await self.audit.log("TXN_COMPLETED", txn.txn_id)
                return txn
            
            elif credit_result.status == "TIMEOUT":
                # UNCERTAINTY: We don't know if credit happened
                # Mark as DEEMED_SUCCESS — settlement will clarify
                txn.state = TransactionState.DEEMED_SUCCESS
                txn.updated_at = datetime.utcnow()
                await self.db.save(txn)
                await self.audit.log("TXN_DEEMED_SUCCESS", txn.txn_id)
                
                # Schedule reconciliation check
                await self.reversal_queue.schedule_check(
                    txn.txn_id,
                    check_at=datetime.utcnow() + timedelta(hours=24)
                )
                return txn
            
            else:
                # Credit FAILED — must reverse the debit
                await self._initiate_reversal(txn)
                return txn
                
        except Exception as e:
            # Unexpected error — check state and recover
            await self.audit.log("TXN_ERROR", txn.txn_id, str(e))
            await self._handle_error(txn, e)
            raise
    
    async def _initiate_reversal(self, txn: Transaction):
        """
        Reverse a failed transaction.
        Credit the debited amount back to sender.
        """
        txn.reversal_initiated = True
        txn.updated_at = datetime.utcnow()
        await self.db.save(txn)
        await self.audit.log("REVERSAL_INITIATED", txn.txn_id)
        
        # Queue for reversal (handled by separate process)
        await self.reversal_queue.enqueue(txn.txn_id)
    
    async def process_reversal(self, txn_id: str):
        """
        Execute reversal — credit money back to sender.
        Called by reversal worker.
        """
        txn = await self.db.get(txn_id)
        
        if txn.reversal_completed:
            return  # Already reversed
        
        # Credit back to sender
        reversal_result = await self.gateway.credit(
            vpa=txn.sender_vpa,
            amount=txn.amount,
            txn_ref=f"REV-{txn.txn_id}",
            timeout=self.credit_timeout
        )
        
        if reversal_result.success:
            txn.state = TransactionState.REVERSED
            txn.reversal_completed = True
            txn.updated_at = datetime.utcnow()
            await self.db.save(txn)
            await self.audit.log("REVERSAL_COMPLETED", txn.txn_id)
        else:
            # Reversal failed — retry later
            # This is a critical alert scenario
            await self.audit.log("REVERSAL_FAILED", txn.txn_id,
                                reversal_result.error)
            await self.reversal_queue.schedule_retry(
                txn.txn_id,
                retry_at=datetime.utcnow() + timedelta(minutes=15)
            )
Transcript
Interviewer

"What about the 'deemed success' state? That seems risky."

You

"Great catch. Here's how reconciliation handles it..."

Deemed success reconciliation

Scenario: We debited Alice, tried to credit Bob, got TIMEOUT

At NPCI level
  • Transaction marked DEEMED_SUCCESS
  • We don't know if Bob got money
  • Settlement file sent to banks includes this transaction
At Bank level (reconciliation, by T+1)
  • Bank compares settlement file with actual credits
  • If credit happened: Mark as COMPLETED
  • If credit NOT happened: Mark as FAILED → Auto-reversal
Timing
  • T+0 — Transaction happens, deemed success
  • T+0/T+1 — Banks reconcile, report actual status
  • By end of T+1 — If the credit failed, the debit is reversed automatically

This is why the RBI's turnaround-time rules say that for a UPI account-to-account transfer where the account is debited but the beneficiary is not credited, the reversal must happen by T+1; after that the bank owes the customer ₹100 per day of delay (for merchant payments the limit is T+5).

In practice, most such cases resolve within hours.

Quick check3 questions · +5 XP each

VPA resolution and atomic transactions

  1. Q1/3 The resolver caches a found VPA for 30 minutes but a "not found" result for only 5 minutes. Why the shorter TTL for misses?

    Answer & explanation

    Answer: B

    Caching "NOT_FOUND" stops repeated lookups of a bad VPA from hitting the bank, while the short TTL limits how long a just-registered VPA looks missing. Timeouts are returned as BANK_TIMEOUT and are not cached as misses at all.

  2. Q2/3 Alice is debited, then the credit call to Bob's bank times out. What should the processor do?

    Answer & explanation

    Answer: D

    A timeout means the credit may or may not have happened. DEEMED_SUCCESS records that uncertainty, and bank reconciliation later confirms it or triggers auto-reversal. Reversing right away is the tempting mistake: if Bob was in fact credited, money would be created out of thin air.

  3. Q3/3 Why does the processor save the transaction record (CREATED, then DEBIT_INITIATED) before calling any bank?

    Answer & explanation

    Answer: A

    Durability before side effects is what makes recovery possible: the system always knows how far a payment got and whether money must be put back. It does not speed anything up; it adds a write, and that cost is accepted for recoverability.

Deep Dive 3: Bank Integration at Scale#

Week 2 concepts: Timeouts, circuit breakers. Week 3 concepts: Message queues.

You

"With 740+ banks, each with different legacy systems, bank integration is a massive challenge."

DiagramBank integration architecture — NPCI → bank gateway
%% title: Bank integration architecture — NPCI → bank gateway
flowchart TD
  subgraph GW["NPCI → Bank gateway"]
    direction TB
    subgraph AL["Bank adapter layer"]
      SA["SBI Adapter<br/>SBI's quirks · retry logic · timeouts"]:::service
      HA["HDFC Adapter<br/>HDFC's quirks · retry logic · timeouts"]:::service
      XA["Axis Adapter<br/>Axis's quirks · retry logic · timeouts"]:::service
      MA["..."]:::service
    end
    subgraph CB["Circuit breaker layer · per-bank breakers prevent cascade failures"]
      CS["SBI: CLOSED · healthy"]:::ok
      CH["HDFC: CLOSED · healthy"]:::ok
      CX["Axis: OPEN · failing, skip for 30s"]:::bad
      CI["ICICI: HALF · testing recovery"]:::queue
      direction TB
      CS ~~~ CX
      CH ~~~ CI
    end
    SEC["SECURE COMMUNICATION LAYER<br/>HTTPS with mutual TLS<br/>Request/Response signing<br/>Encryption of sensitive data<br/>IP whitelisting<br/>Dedicated leased lines to major banks"]:::edge
  end
  SA & HA & XA --> CB
  CB --> SEC

Every bank implements NPCI's standard UPI API (XML messages), but their latency, reliability and error behaviour vary widely. In this design, a per-bank adapter holds each bank's timeouts and retry rules; the format quirks in the code below are illustrative of a generic multi-bank gateway, not of UPI.

pythonCircuitState · 219 lines
# bank/gateway.py

"""
Bank Gateway: Unified interface to 740+ banks.

Each bank is different. This gateway provides a
consistent interface while handling per-bank quirks.
"""

from dataclasses import dataclass
from typing import Dict, Optional
from datetime import datetime, timedelta
from enum import Enum


class CircuitState(Enum):
    CLOSED = "closed"    # Normal operation
    OPEN = "open"        # Bank failing, don't try
    HALF_OPEN = "half"   # Testing if bank recovered


@dataclass
class BankConfig:
    """Configuration for a bank."""
    bank_code: str
    endpoint: str
    timeout_ms: int = 30000
    
    # Circuit breaker settings
    failure_threshold: int = 5
    recovery_timeout_s: int = 30
    
    # Illustrative quirks for a generic multi-bank gateway
    # (UPI itself gives every bank one standard XML API)
    requires_padding: bool = False
    amount_in_rupees: bool = False  # e.g. a partner that wants rupees, not paise
    legacy_xml_format: bool = False


class BankGateway:
    """
    Gateway for all bank operations.
    """
    
    def __init__(
        self,
        http_client,
        bank_configs: Dict[str, BankConfig],
        metrics
    ):
        self.http = http_client
        self.configs = bank_configs
        self.metrics = metrics
        
        # Circuit breakers per bank
        self.circuits: Dict[str, CircuitBreaker] = {
            code: CircuitBreaker(config)
            for code, config in bank_configs.items()
        }
    
    async def debit(
        self,
        bank_code: str,
        account_ref: str,
        amount: int,
        txn_ref: str
    ) -> 'BankResponse':
        """
        Debit an account at a bank.
        
        Amount is in paise (smallest unit).
        """
        config = self.configs.get(bank_code)
        if not config:
            return BankResponse(
                success=False,
                error_code="UNKNOWN_BANK"
            )
        
        # Check circuit breaker
        circuit = self.circuits[bank_code]
        if not circuit.can_execute():
            self.metrics.increment("bank_circuit_open", bank_code)
            return BankResponse(
                success=False,
                error_code="BANK_CIRCUIT_OPEN"
            )
        
        try:
            # Build request (handle bank-specific formats)
            request = self._build_debit_request(
                config, account_ref, amount, txn_ref
            )
            
            # Make the call
            start = datetime.utcnow()
            response = await self.http.post(
                config.endpoint + "/debit",
                json=request,
                timeout=config.timeout_ms / 1000
            )
            latency = (datetime.utcnow() - start).total_seconds() * 1000
            
            # Record metrics
            self.metrics.record_latency("bank_debit", bank_code, latency)
            
            # Parse response
            result = self._parse_response(config, response)
            
            if result.success:
                circuit.record_success()
            else:
                circuit.record_failure()
            
            return result
            
        except TimeoutError:
            circuit.record_failure()
            self.metrics.increment("bank_timeout", bank_code)
            return BankResponse(
                success=False,
                error_code="TIMEOUT",
                status="TIMEOUT"
            )
        except Exception as e:
            circuit.record_failure()
            self.metrics.increment("bank_error", bank_code)
            return BankResponse(
                success=False,
                error_code="BANK_ERROR",
                error_message=str(e)
            )
    
    def _build_debit_request(
        self,
        config: BankConfig,
        account_ref: str,
        amount: int,
        txn_ref: str
    ) -> dict:
        """Build bank-specific request format."""
        
        # Handle amount format (paise vs rupees)
        if config.amount_in_rupees:
            amount_value = amount / 100
        else:
            amount_value = amount
        
        if config.legacy_xml_format:
            # Illustrative: a partner on an older format
            return {
                "xml_payload": self._build_xml(
                    account_ref, amount_value, txn_ref
                )
            }
        
        return {
            "account_reference": account_ref,
            "amount": amount_value,
            "transaction_reference": txn_ref,
            "timestamp": datetime.utcnow().isoformat()
        }


class CircuitBreaker:
    """
    Circuit breaker for bank connections.
    
    Prevents cascade failures when a bank is down.
    """
    
    def __init__(self, config: BankConfig):
        self.config = config
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure: Optional[datetime] = None
        self.success_count = 0
    
    def can_execute(self) -> bool:
        """Check if we can make a request to this bank."""
        if self.state == CircuitState.CLOSED:
            return True
        
        if self.state == CircuitState.OPEN:
            # Check if recovery timeout passed
            if self.last_failure:
                elapsed = (datetime.utcnow() - self.last_failure).total_seconds()
                if elapsed > self.config.recovery_timeout_s:
                    self.state = CircuitState.HALF_OPEN
                    self.success_count = 0
                    return True
            return False
        
        if self.state == CircuitState.HALF_OPEN:
            return True
        
        return False
    
    def record_success(self):
        """Record a successful request."""
        if self.state == CircuitState.HALF_OPEN:
            self.success_count += 1
            if self.success_count >= 3:  # 3 successes to close
                self.state = CircuitState.CLOSED
                self.failure_count = 0
        else:
            self.failure_count = 0
    
    def record_failure(self):
        """Record a failed request."""
        self.failure_count += 1
        self.last_failure = datetime.utcnow()
        
        if self.state == CircuitState.HALF_OPEN:
            # Back to open
            self.state = CircuitState.OPEN
            self.success_count = 0
        elif self.failure_count >= self.config.failure_threshold:
            self.state = CircuitState.OPEN

Deep Dive 4: Security — The Trust Foundation#

Week 9 concepts: Security, authentication, fraud detection.

You

"UPI handles nearly ₹30 trillion a month (₹29.37 lakh crore in September 2026). Security isn't optional — it's existential."

UPI security architecture

Multi-layer security

Layer 1: Device binding
  • The app's UPI registration is bound to the device and SIM (the UPI PIN itself belongs to the bank account)
  • Device identifier (a likely design detail; modern Android no longer exposes the IMEI to apps)
  • SIM binding (mobile number verification)
  • If device changes, re-registration required
Layer 2: Two-factor authentication

Factor 1: Something you HAVE

  • The registered mobile device
  • The SIM card with registered number

Factor 2: Something you KNOW

  • 4-6 digit UPI PIN (set by user)
  • PIN encrypted on device, never transmitted in clear
Layer 3: Encryption
  • HTTPS/TLS for all communication
  • UPI PIN encrypted on the device with a public key (asymmetric RSA, via NPCI's common library); hashing schemes like PBKDF2 do not encrypt
  • PIN verification in bank's HSM (Hardware Security Module)
  • End-to-end encryption for sensitive data
Layer 4: Transaction signing
  • Each transaction signed with digital signature
  • Prevents tampering in transit
  • Non-repudiation for disputes
Layer 5: Real-time fraud detection
  • Velocity checks (too many transactions too fast)
  • Amount anomaly detection
  • Geo-location checks (impossible travel)
  • Behavioral analysis
  • Block list matching

Infrastructure security

NPCI data centers
  • Tier-IV design (Hyderabad is rated for 99.995% availability)
  • Chennai + Hyderabad (geographically separated)
  • HSMs for key storage (FIPS 140-2 Level 3 is the usual bar)
  • Active-Active configuration (this design)
  • N+N redundancy (NPCI, Hyderabad data centre, 2020)
  • Physical security:
    • Biometric access control
    • 24/7 security personnel
    • CCTV surveillance
    • Man-trap entries
Bank connections
  • Dedicated leased lines (not public internet)
  • Mutual TLS authentication
  • IP whitelisting
  • Regular security audits
pythonFraudSignals · 112 lines
# security/fraud_detector.py

"""
Real-Time Fraud Detection for UPI.

Must decide in < 50ms whether to allow a transaction.
"""

from dataclasses import dataclass
from typing import List, Tuple
from datetime import datetime, timedelta


@dataclass
class FraudSignals:
    """Signals used for fraud detection."""
    user_id: str
    device_id: str
    amount: int
    receiver_vpa: str
    
    # Velocity
    txn_count_1hr: int
    txn_count_24hr: int
    total_amount_24hr: int
    
    # Device
    is_new_device: bool
    device_age_days: int
    
    # Behavioral
    is_new_receiver: bool
    typical_amount: int
    typical_time_of_day: List[int]
    current_hour: int
    
    # Location
    device_location: str
    usual_locations: List[str]


class FraudDetector:
    """
    Real-time fraud detection.
    
    Must be FAST (< 50ms) and ACCURATE (low false positives).
    """
    
    def __init__(self, ml_model, rules_engine, blocklist):
        self.model = ml_model
        self.rules = rules_engine
        self.blocklist = blocklist
    
    async def evaluate(
        self,
        signals: FraudSignals
    ) -> Tuple[str, float, List[str]]:
        """
        Evaluate fraud risk.
        
        Returns: (decision, confidence, triggered_rules)
        decision: "ALLOW", "BLOCK", "STEP_UP"
        """
        triggered_rules = []
        
        # RULE 1: Blocklist check (instant)
        if await self.blocklist.is_blocked(signals.user_id):
            return "BLOCK", 1.0, ["USER_BLOCKED"]
        
        if await self.blocklist.is_blocked(signals.device_id):
            return "BLOCK", 1.0, ["DEVICE_BLOCKED"]
        
        # RULE 2: Velocity checks
        if signals.txn_count_1hr > 10:
            triggered_rules.append("HIGH_VELOCITY_1HR")
        
        if signals.txn_count_24hr > 50:
            triggered_rules.append("HIGH_VELOCITY_24HR")
        
        if signals.total_amount_24hr > 100000_00:  # ₹1 lakh in paise
            triggered_rules.append("HIGH_AMOUNT_24HR")
        
        # RULE 3: Amount anomaly
        if signals.amount > signals.typical_amount * 10:
            triggered_rules.append("AMOUNT_ANOMALY")
        
        # RULE 4: New device
        if signals.is_new_device:
            triggered_rules.append("NEW_DEVICE")
            if signals.amount > 10000_00:  # > ₹10,000 on new device
                triggered_rules.append("HIGH_AMOUNT_NEW_DEVICE")
        
        # RULE 5: Unusual time
        if signals.current_hour not in signals.typical_time_of_day:
            triggered_rules.append("UNUSUAL_TIME")
        
        # RULE 6: Location check
        if signals.device_location not in signals.usual_locations:
            triggered_rules.append("UNUSUAL_LOCATION")
        
        # ML model for complex patterns
        ml_score = await self.model.predict(signals)
        
        # Decision logic
        if ml_score > 0.9 or len(triggered_rules) > 3:
            return "BLOCK", ml_score, triggered_rules
        
        if ml_score > 0.7 or len(triggered_rules) > 1:
            # Step-up: require additional verification
            return "STEP_UP", ml_score, triggered_rules
        
        return "ALLOW", 1 - ml_score, triggered_rules
Quick check3 questions · +5 XP each

Bank integration and security

  1. Q1/3 Axis Bank starts hanging on every request. Without a per-bank circuit breaker, what is the biggest risk?

    Answer & explanation

    Answer: D

    Each hung call holds resources until its timeout, so one sick bank can drag down traffic for every bank. An OPEN breaker fails Axis calls fast instead. Double charging is prevented by idempotency, not by the breaker.

  2. Q2/3 ICICI's breaker is HALF_OPEN. Two trial requests succeed, then the third fails. With this gateway code, what happens?

    Answer & explanation

    Answer: B

    In HALF_OPEN any failure sends the breaker back to OPEN; it needs 3 successes in a row to close. The 5-failure threshold applies only when tripping from CLOSED, not while testing recovery.

  3. Q3/3 On a new device, a user sends more than ₹10,000. Only NEW_DEVICE and HIGH_AMOUNT_NEW_DEVICE fire, and the ML score is 0.4. What does the fraud detector return?

    Answer & explanation

    Answer: C

    Two triggered rules exceed the "more than 1" threshold for STEP_UP, even with a low ML score. BLOCK needs a score above 0.9 or more than 3 rules. A low ML score alone does not override the rule count.

Phase 5 · Stage 5 of 7 · 3 min

Phase 5: Scaling and Edge Cases#

Transcript
Interviewer

"What happens during Diwali when everyone is sending money?"

You

"Let's design for 5x spikes during festivals. Here's how..."

Your turn · checkpoint 4not answered

Diwali evening, and the largest bank starts timing out

Traffic is five times normal, packed into 7 to 11 PM, and the biggest bank's latency starts climbing. What did you prepare weeks before? And what stops one slow bank from dragging every other payment down with it?

Think it through, or jot it down. Then compare.

One strong answer

  • Before: pre-scale to around six times the normal peak, warm the VPA caches, tell banks to scale their own systems, and rehearse the runbooks.
  • During: shed what can wait (promotional notifications), rate-limit per user, and queue non-urgent work such as mandate registrations.
  • Isolation: a per-bank circuit breaker. When that bank's failures cross the threshold, payments involving it fail fast with a clear "bank unavailable" message, instead of every request waiting out a timeout and holding threads that other banks' payments need.
  • And rate-limit every API, status checks included. Uncontrolled status-check traffic is what degraded the whole system in April 2025.

Festival Traffic Management#

Diwali scale (design assumption: 5x normal traffic)
Normal day Diwali
Transactions ~800 million ~4 billion
Average TPS ~9,300 ~46,000
Peak TPS ~28,000 ~140,000+

Diwali: Concentrated in evening hours (7 PM - 11 PM)

For reference, the real Diwali-eve 2025 peak was ~754 million transactions in a day, about 13% above October 2025's daily average of 668 million. The 5x here is deliberate headroom, not a forecast.

Preparation (weeks before)
  • Pre-scale infrastructure to ~6x normal peak capacity (≈170K TPS)
  • Warm up caches with popular VPAs
  • Notify banks to scale their systems
  • Extended support staff on standby
  • Runbooks reviewed and tested
During the event
  • Auto-scaling triggers at 60% capacity
  • Non-critical features disabled (promotional notifications)
  • Enhanced monitoring (5-second alert intervals)
  • War room with all bank representatives
  • Direct escalation paths to bank CTOs
Graceful degradation

If overwhelmed:

  • Prioritize smaller transactions (more users served)
  • Rate limit per-user (max 5 txn/minute)
  • Queue non-urgent operations (mandate registrations)
  • Return "Try again in few minutes" vs hard failure

Critical Edge Cases#

Critical edge cases
Edge case 1: Bank System Down

Problem: SBI (the largest remitter bank, roughly a quarter of all UPI debits: 5.2 billion of 20.0 billion in August 2025, per NPCI data) goes down

Impact: Roughly a quarter of transactions fail

Solution:

  • Circuit breaker opens for SBI immediately
  • Return clear error: "SBI temporarily unavailable"
  • Pending transactions queued (if bank supports retry)
  • Status page updated
  • Auto-retry when circuit closes
  • Transactions involving SBI gracefully rejected
Edge case 2: NPCI Switch Partial Failure

Problem: One NPCI data center fails

Impact: 50% capacity lost

Solution:

  • Active-Active setup in Chennai and Hyderabad (this design; NPCI runs data centres in both cities)
  • Traffic automatically routes to healthy DC
  • Health-checked load balancing reroutes traffic (any DNS TTL kept ≤ 10-20 s, or a 60 s TTL alone would break the 30 s RTO)
  • Data replicated synchronously between DCs
  • RPO: 0 (no data loss), RTO: < 30 seconds
Edge case 3: Duplicate Transaction Request

Problem: User's app times out, they retry, but first request succeeded

Impact: Double debit

Solution:

  • Every transaction has idempotency key
  • Generated once when the payment is created (a unique transaction ID stored on the device/PSP) and reused verbatim on every retry; a key built from the retry's own timestamp would never match
  • NPCI checks idempotency before processing
  • If duplicate: return original result
  • No double processing possible
Edge case 4: April 2025 Outage (Real Incident)

What happened:

  • Some PSP banks were calling the "Check Transaction Status" API excessively, without waiting for responses
  • Some re-checked transactions from previous days repeatedly
  • Usage limits on this API weren't enforced at the switch (NPCI added rate limiters afterwards)
  • The switch was flooded: on 12 April 2025 the success rate fell to about 50% for nearly two hours

Lesson learned:

  • Rate limit ALL APIs, not just transaction APIs
  • Enforce guidelines at NPCI firewall, not just bank side
  • Separate critical path APIs from status check APIs
  • Circuit breaker for misbehaving banks
Quick check3 questions · +5 XP each

Scaling and edge cases

  1. Q1/3 A user's app times out and they tap Pay again, but the first request already succeeded. What prevents a double debit?

    Answer & explanation

    Answer: B

    Idempotency is the guarantee: NPCI looks up the key first and, on a duplicate, returns the stored result. Rate limits and breakers manage load; a single retry sits well within them and would still go through.

  2. Q2/3 SBI, the source of roughly a quarter of transactions, goes down. What is the right behavior for the rest of the system?

    Answer & explanation

    Answer: C

    The money lives in SBI accounts, so no other bank can move it; the right move is a fast, clear failure for SBI and isolation for everyone else. NPCI failover does not help, because the switch is healthy; the outage is at the bank.

  3. Q3/3 What was the core lesson of the April 2025 outage?

    Answer & explanation

    Answer: D

    Banks hammered the unlimited "Check Transaction Status" API and it degraded the whole system. The fix is rate limiting all APIs at NPCI and isolating status checks from transactions, not tuning timeouts or caches.

Phase 6 · Stage 6 of 7 · 2 min

Phase 6: Monitoring and Operations#

You

"For a system processing nearly ₹30 trillion a month (September 2026), monitoring isn't optional."

Key Metrics Dashboard#

UPI operations dashboard (illustrative numbers)
Real-time health
Transaction Rate Success Rate Latency (p99)
7,842 TPS 99.2% 1,187 ms
(Target: 10K) (Target: 99.0%) (Target: 2,000ms)
Bank health matrix
Bank TPS Success Latency Circuit Issues
SBI 2,341 98.7% 423ms CLOSED Minor lag
HDFC 1,856 99.8% 287ms CLOSED Healthy
ICICI 1,234 99.5% 312ms CLOSED Healthy
Axis 987 94.2% 892ms HALF-OPEN HIGH LATENCY
Kotak 654 99.1% 345ms CLOSED Healthy
Yes Bank 543 99.4% 298ms CLOSED Healthy
...
Alerts (last 1 hour)
  • 14:32 — Axis Bank latency > 800ms (CRITICAL)
  • 14:28 — SBI error rate 1.3% (Warning)
  • 14:15 — Axis Bank circuit half-open (Info)
  • 13:45 — Traffic spike +20% (Auto-scaled)

SLOs for UPI#

UPI service level objectives (this design; "current" values are illustrative)
SLO 1: Transaction success rate
  • Target: 99.0% of transactions succeed
  • Measurement: Successful / Total (excluding user errors)
  • Current: 99.2%
  • Exclusions:
    • Insufficient balance (user error)
    • Wrong PIN (user error)
    • Account blocked (compliance)
SLO 2: End-to-end latency
  • Target: 99% of transactions complete in < 2 seconds
  • Measurement: Time from request received to response sent
  • Current: p99 = 1.2 seconds
  • Breakdown:
    • NPCI processing: < 300ms
    • Bank response (each of debit and credit): < 700ms
    • Network overhead: < 200ms
SLO 3: Availability
  • Target: 99.9% uptime
  • Measurement: (Total time - Downtime) / Total time
  • Allowed downtime: 8.7 hours/year
SLO 4: Money safety
  • Target: 100% of debited amounts credited or reversed
  • Measurement: No money stuck beyond T+1 (RBI turnaround time for P2P)
  • Current: 99.99% resolved within 24 hours

This is NON-NEGOTIABLE. Error budget = 0

Quick check3 questions · +5 XP each

SLOs for money

  1. Q1/3 The success-rate SLO is 99.0%, yet the money-safety SLO has an error budget of zero. Why the difference?

    Answer & explanation

    Answer: B

    A failed payment costs the user a retry. Money that leaves an account and is neither credited nor reversed breaks the one promise a payment system makes, so every such case must be resolved, which is why reconciliation and T+1 reversal exist.

  2. Q2/3 The success-rate SLO excludes insufficient balance and wrong PIN. Why?

    Answer & explanation

    Answer: C

    An SLO should move when the system misbehaves. Counting users' own declined payments would burn the error budget on things engineers can't fix and blur the signal they can.

  3. Q3/3 The bank health matrix shows Axis Bank at 94.2% success, 892 ms latency and a HALF-OPEN circuit. What is the switch doing for Axis right now?

    Answer & explanation

    Answer: B

    HALF-OPEN is the probe state: after the open timeout, a few trial calls go through. Enough consecutive successes close the breaker; any failure sends it back to OPEN.

Final challenge · Stage 7 of 7 · 2 min

Final Challenge: The Flash-Sale Merchant#

The interviewer leans back. "Last one. A curveball."

Interviewer

"A huge online sale opens at noon, and one merchant's payment address receives 50,000 payments a minute for an hour. Everything else stays normal. Where does your design get hot, and what do you change?"

The flash-sale brief
  • One payee VPA, 50,000 incoming payments a minute, for an hour
  • Payers are spread across hundreds of banks; the merchant banks with one
  • Every payer expects the usual few-second confirmation
  • Payers who time out will tap Pay again
Your turn · checkpoint 5not answered

Find the hot spots before you fix anything

Trace a single payment through your design and mark every place where all 50,000 a minute meet the same key, row or queue. Which of those are reads, which are writes, and what would you do for each?

Think it through, or jot it down. Then compare.

One strong answer

  • The payee lookup (a read): every payment resolves the same VPA. That's a hot key, and the easy one: it's already in the switch's VPA cache, and a short-lived local copy on each switch node takes it off the shared cache entirely.
  • The merchant's account (the real write hotspot): every credit updates the same balance in the merchant's bank, so the bank serialises them on one row. Fix it by spreading credits: append ledger entries and compute the balance asynchronously, or credit several sub-accounts and roll them up.
  • Duplicates: retries come from payers who timed out, so every retry must carry the original transaction ID, and the switch returns the original result instead of debiting twice.
  • Guards that must not misfire: per-user rate limits and velocity checks are for payers. Applied to the receiving merchant, they would reject honest customers.
  • Notifications: batch or aggregate the merchant's confirmations rather than pushing 50,000 separate alerts a minute to one device.
Quick check4 questions · +5 XP each

Final challenge · Quest check

  1. Q1/4 50,000 payments a minute to one merchant is about how many per second?

    Answer & explanation

    Answer: B

    50,000 ÷ 60 ≈ 833 per second. That's a small share of the switch's ~28,000 TPS peak, which is the point: the problem isn't total volume, it's that all of it lands on one key and one account.

  2. Q2/4 Where is the hardest hot spot in the flash-sale scenario?

    Answer & explanation

    Answer: B

    Reads of one key are cheap to cache. Writes to one balance row have to be serialised, so they are the real bottleneck. The switch's own transaction rows are spread across partitions by transaction ID, and the payers' debits are spread over many accounts and banks.

  3. Q3/4 A payer's app times out after they tap Pay, and they tap again. What prevents a double debit?

    Answer & explanation

    Answer: C

    The idempotency key is created once, with the payment, and reused on every retry. A key built from the retry's own timestamp would never match, and an amount-based rule would reject genuine repeat purchases.

  4. Q4/4 Why should a "5 payments per minute" velocity limit apply to payers and not to the receiving merchant?

    Answer & explanation

    Answer: B

    Velocity rules model normal behaviour for a role. Five payments a minute is a lot for one person to send and nothing for a busy merchant to receive, so the same threshold on the payee side would turn a fraud guard into an outage.

Wrap-up · 5 min

Interview Conclusion#

Transcript
Interviewer

"Impressive depth. A few rapid-fire questions:"

Interviewer

"Why didn't India just adopt an existing system like Visa/Mastercard?"

You

"Three reasons:

  1. Cost: Card payments typically cost merchants ~1-3% (merchant discount rate). UPI is nearly free.
  2. Inclusion: Cards need plastic issuance and POS terminals (and credit cards need credit checks). UPI needs only a phone.
  3. Control: Critical financial infrastructure shouldn't depend on foreign companies.

The result: UPI enabled the chai vendor to accept digital payments for a ₹10 tea."

Interviewer

"What's the biggest technical achievement of UPI?"

You

"Interoperability without centralized money holding. NPCI routes transactions but never touches the money. This means:

  • No counterparty risk (NPCI can't go bankrupt with your money)
  • Banks remain the regulated entities
  • Scales horizontally (NPCI is just a switch)
  • Any app works with any bank

This architecture is why countries worldwide are studying UPI."

Interviewer

"If you were to improve UPI today, what would you change?"

You

"Based on the April 2025 outage:

  1. Stricter rate limiting at NPCI level, not trusting banks to self-regulate
  2. Better isolation between critical transaction APIs and status check APIs
  3. More granular circuit breakers — per-API, not just per-bank
  4. Chaos engineering — regularly test failure scenarios in production"

Summary: Concepts Applied from 10-Week Course#

Concepts from 10-week course in UPI design
  • Week 1: Data at scale
    • Partitioning — VPAs partitioned by bank suffix
    • Replication — Multi-DC active-active setup
    • Read optimization — VPA caching at NPCI level
  • Week 2: Failure-first design
    • Timeouts — Strict timeouts for bank calls (30s)
    • Circuit breakers — Per-bank failure isolation
    • Idempotency — Transaction idempotency keys
    • Retries — Smart retry with exponential backoff
  • Week 3: Messaging & async
    • Transactional outbox — Audit logging
    • Dead letter queues — Failed reversal handling
    • Event streaming — Transaction events for reconciliation
  • Week 4: Caching
    • VPA resolution caching
    • Bank configuration caching
    • Negative caching — Non-existent VPAs
  • Week 5: Consistency & coordination
    • Distributed transactions — Debit-then-credit with rollback
    • State machine — Transaction lifecycle management
    • Exactly-once semantics — Idempotency guarantees
  • Week 9: Security & compliance
    • Multi-factor authentication — Device + PIN
    • Encryption — Public-key PIN encryption, HSM-based PIN verification
    • Fraud detection — Real-time ML scoring
    • Audit trail — Complete transaction logging
  • Week 10: Production readiness
    • SLOs — Success rate, latency, availability targets
    • Observability — Per-bank health dashboards
    • Capacity planning — Festival traffic handling
    • Incident management — April 2025 outage learnings

Why UPI Matters#

Why UPI is a marvel of engineering
Scale
  • ~49% of the world's real-time payment volume (2023)
  • More than Visa + Mastercard combined (in India)
  • ~800 million transactions DAILY
Inclusion
  • Works on ₹3,000 smartphones
  • Works on 2G networks
  • ₹1 transactions viable (no minimums)
  • Brought hundreds of millions of first-time users into digital payments
Cost
  • Zero cost to consumers
  • Near-zero cost to small merchants
  • Saved billions in card network fees
Innovation
  • VPA system (email for money)
  • Interoperable (any app, any bank)
  • Open standard (countries can adopt)
  • Built on existing bank infrastructure
Global impact
  • 11 countries accepting UPI (August 2026)
  • More countries in talks to adopt it
  • Model for BIS cross-border payment initiatives

"UPI proved that with the right architecture, a developing nation can leapfrog decades of financial infrastructure and build something the developed world envies."

Self-Assessment Checklist#

After studying this case study, you should be able to:

Architecture:

Architecture0/3 ready
  • Explain the three-layer UPI architecture (Apps → NPCI → Banks)
  • Design a VPA resolution system with caching
  • Implement atomic transactions with rollback capability

Distributed Systems:

Distributed Systems0/3 ready
  • Handle partial failures in multi-party transactions
  • Implement circuit breakers for unreliable dependencies
  • Design idempotency for payment systems

Scale:

Scale0/3 ready
  • Calculate infrastructure needs for billion-transaction systems
  • Plan for bursty traffic (festivals, events)
  • Implement graceful degradation under load

Security:

Security0/3 ready
  • Design multi-factor authentication for payments
  • Implement real-time fraud detection
  • Understand HSM-based PIN verification

Operations:

Operations0/3 ready
  • Define meaningful SLOs for payment systems
  • Monitor multi-party systems (NPCI + 740 banks)
  • Learn from production incidents (April 2025 outage)

Sources#

Statistics and Data:

Architecture and Technical Details:

Infrastructure and Security:

Settlement and Operations:

Outage Analysis (April 2025):

Further Reading#

Official Documentation:

Engineering Blogs and Technical Deep Dives:

Research Papers and Reports:

  • Paper

    BIS Papers No. 152: Faster Digital Payments - Global and Regional Perspectives (India Chapter)

  • Paper

    NPCI White Papers: Available on NPCI website

  • Read

    RBI Annual Reports: Digital payments statistics and trends

News and Industry Analysis:

Video Resources:

  • Talk

    NPCI YouTube Channel: Official explainers and announcements

  • Talk

    System Design Interview Videos: Search "UPI System Design" on YouTube

Books:

  • Book

    "Designing Data-Intensive Applications" by Martin Kleppmann - Foundational concepts

  • Book

    "System Design Interview" by Alex Xu - Interview preparation with similar patterns

Related Systems to Study:

  • Read

    PIX (Brazil): Similar instant payment system

  • Read

    FedNow (USA): US real-time payment system

  • Read

    SEPA Instant (Europe): European instant payments

"A payment system that serves a billion people, handles trillions in transactions, and costs nothing to use. This is what engineering at scale looks like."

this lesson in shorts · 3

play them all →

One idea from this lesson in about a minute each, made for your phone.

End of Side quest 01 · India's UPI
Clear the questFinish UPI.Stages clear as you read them. Marking complete unlocks after ~7 more min of reading.
  • 0/7stages cleared

Follow-up drills

0 of 10 practised

You presented India's UPI. Now the interviewer pushes back. Answer each follow-up out loud or in the box, then reveal a strong answer.

Follow-up 1 / 10 · scale & numbers

“You said ~28,000 to 46,000 TPS at peak. Walk me from 800 million a day to that number. Is that what you size the switch for?”

Pushes on · sizing on transactions instead of the work each one causes

Revealing marks it practised.
design sheet · 6 diagrams · pdfRevise the whole design in ten minutesThe numbers, architecture, sequence, class and state diagrams, decisions and failure modes of this quest on a few printable pages.open the sheet →

Go deeper · reading trail

all reading →

Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.

RazorpayBlog post · ~10 min

Five Years of Kafka at Razorpay's UPI Switch

Kshitij Nawandar · 2026
What
The story of the messaging layer inside Razorpay's UPI Switch, which talks to NPCI for most of its UPI volume.
Key idea
UPI is asynchronous, so a message bus sits at the heart of the switch; SQS fan-out gave way to Kafka, then topics were consolidated when partitions exploded.
Why here
What 'bank integration at scale' looks like inside a payment service provider: every UPI callback is an event fanned out to NPCI updates, merchant callbacks and the warehouse.
taught here: Bank integration at scale
PhonePeBlog post · ~9 min

Demystifying TStore: The Backbone of Billions of Transactions at PhonePe

Arnab Bir, Tushar Naik · 2024
What
PhonePe's transaction store behind the green success screen and the payment history feed.
Key idea
Every service in a payment flow writes to TStore, which keeps one consolidated record per transaction and ingests around 45,000 writes a second at peak.
Why here
The UPI app side of this quest: a pending payment must show one consistent status even while the bank and NPCI are still answering.
taught here: Transaction status and history

New lessons by email

New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.

Discussion · …

Public: everyone reading this lesson sees comments. Be kind and keep it on the material.

Sign in to ask a question or answer one. Reading stays open to everyone.
loading the discussion…