Want it harder? Hide the model answer and work each phase in a scratchpad against its time budget, then reveal and score yourself. Or simply read on: the checkpoints below work either way.
How Do You Make 777 Million People Feel Like You Built an App Just for Them?
Imagine this challenge: You need to serve 100+ million songs to 777 million users across 184 markets. Every user expects instant playback—less than 200ms from tap to music. Every user expects recommendations that feel eerily personal. And you need to do this with an engineering team that pioneered the "squad" model of autonomous teams.
This is Spotify—and it's not just a music app. It's a masterclass in personalization at scale, microservices architecture, and building developer platforms that thousands of engineers can use without stepping on each other's toes.
Users
- Monthly Active Users: 777 Million (Q2 2026)
- Premium Subscribers: 300 Million (Q2 2026)
- Conversion Rate: ~39%
Content
- Songs: 100+ Million
- Podcasts: ~8 Million titles
- Audiobooks: 800,000+ (select regions)
- Playlists: 8+ Billion (user-created, 2024)
Engagement
- Average listening time: ~114 minutes/day (third-party estimate; Spotify doesn't publish one)
- Discover Weekly: 100B+ tracks streamed since its 2015 launch
- New artist discoveries from personalized recommendations: ~1 in 3
Infrastructure
- Markets: 184
- Employees: ~7,300
- Microservices: 2,000+ backend services
- Daily events collected: 1+ Trillion (2024)
Business
- 2025 Revenue: €17.2 Billion
- Market Share: 31.4% of global music subscribers (end of 2025, MIDiA)
- Royalties paid (2025): $11 Billion+ to rights holders
This is the system we'll design today—and discover how Spotify built the world's most personalized audio experience.
Spotify serves 777 million monthly users from 100 million+ tracks, starting playback in under 200 ms: 5 billion streams a day, ~175,000 a second at peak, ~5.5 Tbps. The Stream Service authorizes once and returns a signed URL, audio comes from the CDN edge (origin only on a miss) and play events go asynchronously to Pub/Sub. The deep dives cover hybrid recommendations and cold start, device, edge and origin caching, the event pipeline and breakers, then album drops, licensing and tunnels, and exactly-once play counts for royalties.
The Interview Begins#
You walk into the interview room. The interviewer smiles and gestures to the whiteboard.
"Thanks for coming in. Today we're going to design a music streaming service like Spotify. I'm interested in how you think about scale, personalization, and delivering a seamless listening experience. Please think out loud—this is collaborative."
They write on the whiteboard:
Requirements
- Stream audio to hundreds of millions of users globally
- Near-instant playback (< 200ms to first audio)
- Highly personalized recommendations
- Search across 100M+ tracks
- Support both free (ad-supported) and premium tiers
- Handle massive catalog: songs, podcasts, audiobooks
"Take a few minutes to think about this, then walk me through your approach. We have about 45 minutes."
Phase 1: Requirements Clarification (5 minutes)#
Before diving in, you take a breath and start asking questions.
Your Questions#
"Before I start designing, I'd like to clarify a few requirements. First, what's our target scale—how many concurrent users should we support?"
"At peak, we might have 50 million concurrent listeners. Average is around 20-30 million."
"For playback latency, what's acceptable? I want to understand the user experience bar."
"Users tap a song and expect music within 200 milliseconds. No buffering during playback except on very poor connections."
"How personalized are the recommendations? Are we talking basic 'similar artists' or deeply personal like Discover Weekly?"
"Deeply personal. We want users to feel the app knows their taste better than they do. Recommendations should work even for new users with minimal listening history."
"What about offline playback? Do premium users expect to download music?"
"Yes, premium users can download. That's a key differentiator from the free tier."
"Perfect. Let me summarize the requirements as I understand them."
Functional Requirements#
1. Audio playback
- Stream songs, podcasts, and audiobooks
- Support multiple quality levels (96kbps to 320kbps)
- Adaptive bitrate based on network conditions
- Offline download for premium users
- Gapless playback between tracks
2. Discovery & search
- Full-text search across songs, artists, albums, podcasts
- Typo tolerance and autocomplete
- Browse by genre, mood, activity
- Personalized recommendations
3. User features
- Create and manage playlists
- Follow artists and friends
- Like/save songs to library
- View listening history
4. Monetization
- Free tier with ads
- Premium tier (ad-free, offline, higher quality)
- Family and student plans
Non-Functional Requirements#
1. Scale
- 50 million concurrent users (peak)
- 100+ million songs in catalog
- 500+ billion events/day for analytics
- 5+ billion streams/day
2. Latency
- Playback start: < 200ms
- Search results: < 100ms
- API responses: < 50ms (p99)
3. Availability
- 99.99% uptime for playback
- Graceful degradation (recommendations can fail, playback must not)
4. Data
- Strong consistency for user data (playlists, library)
- Eventual consistency acceptable for recommendations
Phase 2: Back of the Envelope Estimation (5 minutes)#
"Let me work through the numbers to understand the scale."
Before the numbers: how heavy is audio?
250 million people listen each day, about 20 songs each, and a song at 160 kbps is about 4 MB. How many streams start per second on average and at a 3× peak? How much egress is that, and how does it compare with video?
One strong answer
- Streams: 250M × 20 = 5 billion a day, ≈ 58,000 per second on average and ~175,000 at peak.
- Data: 5B × 4 MB = 20 PB a day, about 1.85 Tbps on average and ~5.5 Tbps at peak.
That's roughly a tenth of a video service's egress. Bandwidth is manageable with an ordinary CDN; the hard problems are the 200 ms start and personalisation for hundreds of millions of people.
Traffic Estimation#
- Base numbers
- Daily Active Users: 250 million
- Songs per user per day: 20 songs (average)
- Average song length: 3.5 minutes
- Average file size (160 kbps): ~4 MB per song
- Daily calculations
- Total streams/day: 250M × 20 = 5 billion streams
- Streams per second: 5B ÷ 86,400 = ~58,000 streams/sec
- Peak (3x average): ~175,000 streams/sec
- Bandwidth
- Data per stream: ~4 MB
- Daily data transfer: 5B × 4 MB = 20 PB/day
- Average bandwidth: 20 PB × 8 ÷ 86,400 s ≈ 1.85 Tbps
- Peak bandwidth (3x): ~5.5 Tbps
Storage Estimation#
- Song catalog
- Total songs: 100 million
- Average song (all quality levels): ~25 MB (multiple encodings)
- Total audio storage: 100M × 25 MB = 2.5 PB
- User data
- Users: 777 million
- Per user (playlists, library): ~50 KB average
- Total user data: ~39 TB
- Event data (analytics)
- Events per day: 500 billion
- Daily event storage: ~250 TB/day
- Yearly (compressed): ~10 PB
Key Metrics Summary#
Traffic
- Peak streams: 175,000 /second
- Daily streams: 5 billion
- API requests: ~500,000 /second (peak)
Storage
- Audio catalog: 2.5 PB
- User data: 39 TB
- Event data: ~10 PB/year
Bandwidth
- Peak egress: ~5.5 Tbps
- Daily transfer: 20 PB
Requirements and estimation
Q1/3 250M daily active users each play about 20 songs a day, and peak is 3x the average. Roughly what peak stream rate should you design for?
Answer & explanation
Answer: B
250M x 20 = 5B streams/day, and 5B / 86,400 is about 58,000/sec on average. Tripling that for peak gives ~175,000/sec; ~58,000 is only the average.
Q2/3 Which data must be strongly consistent, and which can be eventually consistent?
Answer & explanation
Answer: D
A song you just added to a playlist should show up on your other device, so user data needs strong consistency. A Discover Weekly list that's a few hours behind your latest plays hurts no one.
Q3/3 Which storage item keeps growing and soon outweighs the whole audio catalog?
Answer & explanation
Answer: A
The catalog is about 2.5 PB and grows slowly, and user data is tiny. The analytics events add roughly 10 PB every year, so the event pipeline drives long-term storage and processing cost.
Phase 3: High-Level Design (10 minutes)#
"Now let me sketch out the high-level architecture."
Sketch it first: what sits on the play path?
A listener taps a song. What must happen before the first note plays, and what can safely happen afterwards?
One strong answer
- Before the first note: check the session and entitlement (tier, licensing in this country), pick the encoding, issue a signed URL, and fetch audio from the CDN edge. Often the client already has the track cached or prefetched, so even that fetch is skipped.
- After: the play event goes asynchronously to the event pipeline, which feeds listening history, royalty counts and recommendations.
- Never on the play path: recommendations. The requirement says it directly: recommendations can fail, playback must not.
System Architecture#
%% title: Spotify high-level architecture
flowchart TD
subgraph CL["Clients"]
direction TB
IOS@{ icon: "tech:apple", label: "iOS" }
AND@{ icon: "tech:android", label: "Android" }
DSK@{ icon: "sd:desktop", label: "Desktop" }
WEB@{ icon: "sd:browser", label: "Web" }
end
subgraph GCP["Google Cloud Platform"]
direction TB
CDN@{ icon: "sd:cdn", label: "CDN (e.g. Akamai/Fastly)<br/>Audio delivery, static assets, caching" }
APIG["API Gateway<br/>Rate limiting, auth, routing, load balancing"]:::edge
subgraph MS["Microservices layer"]
direction TB
US["User Service"]:::core
CS["Catalog Service"]:::core
SS["Search Service"]:::core
ST["Stream Service"]:::hot
PL["Playlist Service"]:::core
RC["Reco Service"]:::core
SO["Social Service"]:::core
AD["Ad Service"]:::core
US ~~~ PL
CS ~~~ RC
SS ~~~ SO
ST ~~~ AD
end
subgraph DL["Data layer"]
direction TB
PG@{ icon: "tech:postgresql", label: "Postgres (Users)" }
CA@{ icon: "tech:cassandra", label: "Cassandra (Events)" }
RD@{ icon: "tech:redis", label: "Redis (Cache)" }
BQ@{ icon: "tech:bigquery", label: "BigQuery (Analytics)" }
GCS@{ icon: "tech:gcs", label: "GCS (Audio)" }
EL@{ icon: "tech:elasticsearch", label: "Elastic (Search)" }
BT@{ icon: "tech:bigtable", label: "Bigtable (Reco)" }
PG ~~~ GCS
CA ~~~ EL
RD ~~~ BT
end
subgraph ES["Event streaming"]
direction TB
PS@{ icon: "tech:pubsub", label: "Pub/Sub (Events)" }
DF@{ icon: "tech:dataflow", label: "Dataflow (Process)" }
MLP["ML Pipeline"]:::service
end
APIG --> MS
MS ~~~ DL
DL ~~~ ES
end
CL ==> CDN
CL --> APIG
class IOS,AND,DSK,WEB client
class CDN,RD cache
class PG,CA,BQ,GCS,EL,BT store
class PS queue
class DF serviceData Flow: Playing a Song#
%% title: Playback flow — playing a song
sequenceDiagram
autonumber
participant C as Client
participant G as API Gateway
participant S as Stream Service
participant CDN as CDN edge
participant O as GCS origin
participant P as Pub/Sub
C->>G: TAP TO PLAY
G->>S: AUTHORIZATION
Note over S: Checks user subscription tier<br/>Regional availability · Content licensing
S-->>C: GENERATE SIGNED URL · token, expires in 1 hour
Note over C,S: https://cdn.spotify.com/audio/abc123?token=xyz&expires=...
C->>CDN: STREAM FROM CDN
alt cached (99%+ hit rate for popular)
CDN-->>C: audio · adaptive bitrate based on network
else cache miss
CDN->>O: fall back to GCS origin
O-->>CDN: audio
CDN-->>C: audio · adaptive bitrate based on network
end
C-)P: TRACK EVENTS · play/skip events
Note over P: → Analytics pipeline → Recommendations
Note over C,P: TOTAL LATENCY: < 200ms to first audioHigh-level design
Q1/3 Why does the Stream Service return a signed, expiring URL instead of the CDN asking the backend about every request?
Answer & explanation
Answer: C
Authorization happens once, at play time. After that the CDN can verify the signature itself, which keeps the backend off the hot path for every audio byte, and the one-hour expiry keeps leaked links from staying useful.
Q2/3 A user plays an obscure track that isn't cached at the edge. What happens?
Answer & explanation
Answer: A
On a cache miss the CDN goes back to the GCS origin. Popular tracks hit the cache over 99% of the time, while long-tail tracks sometimes pay the origin round trip.
Q3/3 The client sends play and skip events to Pub/Sub asynchronously. Why not record them synchronously before playback starts?
Answer & explanation
Answer: D
Tracking feeds analytics and recommendations, but the music doesn't need it to start. Sending events asynchronously protects the latency budget for what the user is waiting on.
Phase 4: Deep Dives (20 minutes)#
"Let's dive deeper. Tell me about the recommendation system."
Cold start, twice
Collaborative filtering learns from what people play. What do you recommend to a brand-new user, and how does a song released this morning, played by nobody yet, ever get recommended?
One strong answer
- New users: onboarding picks (a few favourite artists), locale and demographic signals, editorial and popular playlists, then learn fast from the first handful of plays and skips.
- New songs: content-based signals. Audio analysis turns the track into a feature vector (tempo, energy, mood) so it can be matched to listeners of similar-sounding music; add metadata, the artist's existing fans and editorial placement.
- Then hand over: once real plays accumulate, collaborative filtering takes over. Keeping a share of recommendations for exploration is what lets new music collect those plays.
Deep Dive 1: Recommendation System#
The Problem#
Scale
- 777 million users with unique tastes
- 100+ million songs to choose from
- Generate personalized playlists for each user
Cold start problem
- New users: No listening history
- New songs: No user interactions
Exploration vs exploitation
- Recommend what users will like (exploitation)
- Introduce new music (exploration)
The Solution: Hybrid Recommendation Architecture#
%% title: Three recommendation models flowchart TD CF["COLLABORATIVE FILTERING<br/>'Users like you also liked...'<br/>Matrix factorization"]:::service CB["CONTENT BASED<br/>'Songs with similar tags'<br/>NLP on blogs, reviews, titles"]:::service AA["AUDIO ANALYSIS<br/>'Songs that sound similar'<br/>CNN on audio spectrograms"]:::service EN["ENSEMBLE MODEL<br/>Combines all signals with learned<br/>weights from A/B testing"]:::core CF & CB & AA ==> EN
Collaborative Filtering#
User-Song Matrix (sparse)
| Song1 | Song2 | Song3 | ... | Song100M | |
|---|---|---|---|---|---|
| User1 | 5 | 0 | 3 | ... | 2 |
| User2 | 0 | 4 | 0 | ... | 0 |
| User3 | 3 | 0 | 5 | ... | 4 |
| ... |
Decompose into
- User vectors: 777M × 128 dimensions
- Song vectors: 100M × 128 dimensions
Similarity = dot_product(user_vector, song_vector)
For Discover Weekly
- Find users with similar taste profiles
- Find songs those users love that you haven't heard
- Rank by predicted engagement
Audio Analysis (CNN)#
Input: Audio spectrogram
Process
Raw audio → Mel spectrogram → CNN → 128-dim feature vector
Output features
- Tempo, key, mode
- Danceability, energy, valence
- Acousticness, instrumentalness
Use case: Cold start for new songs
- Works even with zero user interaction data
- Powers "song radio" feature
Implementation#
# recommendations/discover_weekly.py
"""
Discover Weekly Generation
Hybrid recommendation combining collaborative filtering,
content-based filtering, and audio analysis.
"""
from dataclasses import dataclass
from typing import List, Dict, Set
import numpy as np
@dataclass
class UserProfile:
user_id: str
embedding: np.ndarray # 128-dim taste vector
top_artists: List[str]
top_genres: List[str]
listening_history: Set[str]
class DiscoverWeeklyGenerator:
"""
Generates personalized weekly playlists.
Applies concepts:
- Week 5: Eventual consistency for recommendations
- Week 8: Batch processing pipeline
"""
def __init__(self, user_store, song_store, similar_users_index):
self.user_store = user_store
self.song_store = song_store
self.similar_users_index = similar_users_index
# Weights from A/B testing
self.weights = {
'collaborative': 0.4,
'content': 0.3,
'audio': 0.2,
'popularity': 0.1,
}
self.playlist_size = 30
self.exploration_ratio = 0.2
async def generate_playlist(self, user_id: str) -> List[str]:
"""Generate Discover Weekly for a user."""
user = await self.user_store.get_profile(user_id)
# Get candidates from collaborative filtering
collab_candidates = await self._get_collaborative_candidates(user)
# Get candidates from content similarity
content_candidates = await self._get_content_candidates(user)
# Merge and score
all_candidates = self._merge_candidates(
collab_candidates, content_candidates
)
scored = await self._score_candidates(user, all_candidates)
# Filter already-heard songs
filtered = [
(song_id, score) for song_id, score in scored
if song_id not in user.listening_history
]
# Exploitation: top songs
exploit_count = int(self.playlist_size * (1 - self.exploration_ratio))
exploit_songs = [s for s, _ in filtered[:exploit_count]]
# Exploration: outside comfort zone
explore_songs = await self._get_exploration_tracks(
user,
count=self.playlist_size - exploit_count,
exclude=set(exploit_songs)
)
return self._shuffle_with_variety(exploit_songs + explore_songs)
async def _get_collaborative_candidates(
self, user: UserProfile
) -> Dict[str, float]:
"""Find songs from similar users."""
similar_users = await self.similar_users_index.find_similar(
user.embedding, k=1000
)
candidates = {}
for similar_user_id, similarity in similar_users:
loved_songs = await self.user_store.get_loved_songs(
similar_user_id, limit=100
)
for song_id, engagement in loved_songs:
score = similarity * engagement
candidates[song_id] = candidates.get(song_id, 0) + score
return candidates
async def _score_candidates(
self, user: UserProfile, candidates: Dict[str, float]
) -> List[tuple]:
"""Score using ensemble of all signals."""
scored = []
for song_id, base_score in candidates.items():
song = await self.song_store.get_song(song_id)
# Combine signals
final_score = (
self.weights['collaborative'] * base_score +
self.weights['content'] * self._content_similarity(user, song) +
self.weights['audio'] * self._audio_similarity(user, song) +
self.weights['popularity'] * song.popularity_score
)
scored.append((song_id, final_score))
return sorted(scored, key=lambda x: x[1], reverse=True)Deep Dive 2: Audio Streaming#
"How do you achieve < 200ms playback latency?"
Under 200 milliseconds from tap to music
List the places the audio for a tap can come from, fastest first, and the tricks that make most taps feel instant.
One strong answer
- The device: recently played tracks, offline downloads, and the next tracks in the queue, prefetched while the current one plays. For most taps nothing goes over the network.
- The CDN edge: a small share of the catalogue accounts for most plays, so popular tracks are essentially always at the edge, with access tokens validated there and no backend round trip.
- The origin: only on an edge miss, for the long tail.
- Start light, then upgrade: begin with a lower bitrate or a small first chunk so audio starts at once, then switch up.
Multi-Layer Caching#
- Layer 1: Client cache
- Recently played songs
- Prefetched next songs
- Offline downloads (Premium)
- 1-10 GB local storage
- Layer 2: CDN edge (e.g. Akamai/Fastly)
- Edge locations worldwide
- Top 20% of songs always cached (80% of plays)
- 99%+ cache hit rate for popular content
- Token validation at edge
- Layer 3: Origin (GCS)
- All 100M+ songs
- Multiple quality encodings per song
- Multi-region replication
The Stream Service checks tier, region and licensing once and returns a URL signed for 1 hour; the CDN edge verifies the token itself and serves popular tracks from cache (99%+ hit rate) in under 200 ms. A long-tail track missing at the edge is fetched from the GCS origin, so its first play is slower, and a prefetched next track plays straight from the client cache with no network wait.
Audio Quality Tiers#
| Tier | Bitrate | Codec | Use Case |
|---|---|---|---|
| Low | ~24 kbps | Not published | Extreme data saving |
| Normal | ~96 kbps | Ogg Vorbis | Standard / mobile data |
| High | ~160 kbps | Ogg Vorbis | Free tier max |
| Very High | ~320 kbps | Ogg Vorbis | Premium only |
| Lossless | up to 24-bit/44.1 kHz (~1,000+ kbps) | FLAC | Premium (rolled out Sept–Oct 2025) |
Spotify's support page gives the bitrates as approximate and names FLAC for Lossless; its web player streams AAC instead (128 kbps Free, 256 kbps Premium). The Ogg Vorbis column reflects the format Spotify's apps have long used, not a per-tier list Spotify publishes.
Why Ogg Vorbis?
- Open source (no licensing fees)
- Better quality than MP3 at same bitrate
Implementation#
# streaming/playback_service.py
"""
Playback Authorization and URL Signing
Optimized for < 30ms authorization latency.
"""
from dataclasses import dataclass
from typing import List
from datetime import datetime, timedelta, timezone
import hmac
import hashlib
import base64
@dataclass
class PlaybackResponse:
stream_url: str
expires_at: datetime
quality: str
prefetch_urls: List[str]
class PlaybackService:
"""
Handles playback authorization.
Applies concepts:
- Week 4: Multi-tier caching
- Week 2: Timeout management
"""
def __init__(self, rights_service, catalog_service, cdn_config):
self.rights = rights_service
self.catalog = catalog_service
self.cdn_base = cdn_config['base_url']
self.signing_key = cdn_config['signing_key']
async def get_playback_url(
self, user_id: str, song_id: str, quality: str
) -> PlaybackResponse:
"""Generate signed streaming URL."""
import asyncio
# Parallel checks for speed
rights_ok, song = await asyncio.gather(
self.rights.check(user_id, song_id),
self.catalog.get_song(song_id)
)
if not rights_ok:
raise PlaybackNotAllowedError()
# Generate signed URL
file_id = f"{song_id}_{quality}.ogg"
expires = datetime.now(timezone.utc) + timedelta(hours=1)
url = self._sign_url(file_id, expires)
# Prefetch hints for next songs
prefetch = await self._get_prefetch_urls(user_id, song_id, quality)
return PlaybackResponse(
stream_url=url,
expires_at=expires,
quality=quality,
prefetch_urls=prefetch
)
def _sign_url(self, file_id: str, expires: datetime) -> str:
"""HMAC-signed CDN URL."""
expires_ts = int(expires.timestamp())
message = f"/audio/{file_id}:{expires_ts}"
sig = hmac.new(
self.signing_key.encode(),
message.encode(),
hashlib.sha256
).digest()
sig_b64 = base64.urlsafe_b64encode(sig).decode()
return f"{self.cdn_base}/audio/{file_id}?expires={expires_ts}&sig={sig_b64}"Recommendations and audio delivery
Q1/3 A song released this morning has zero plays. Which model can still place it in recommendations?
Answer & explanation
Answer: B
Collaborative filtering needs interactions the song doesn't have yet, and its popularity is zero. Audio features like tempo, energy and danceability are computed from the audio itself, which is how new songs get around cold start.
Q2/3 Discover Weekly has 30 tracks and an exploration ratio of 0.2. How many tracks deliberately come from outside the user's comfort zone?
Answer & explanation
Answer: C
The generator takes 30 x (1 - 0.2) = 24 top-scored "exploit" tracks, and the remaining 6 are exploration picks. The exploration share keeps the playlist from turning into a filter bubble.
Q3/3 How do prefetched next songs help a user who skips to the next track?
Answer & explanation
Answer: A
The client cache is the fastest layer. When the playback response includes prefetch URLs, the next track's audio can be on the device before the user taps, which makes near-instant skips and gapless playback possible.
Deep Dive 3: Microservices & Backstage#
"How do you manage 2,000+ microservices?"
Backstage Developer Portal#
Software catalog
Every service registered with:
- Owner (which squad)
- Description and documentation
- API specifications
- Dependencies
- Health metrics
Core features
- Software Templates (Scaffolder) — New service wizard
- TechDocs — Docs as code
- Search — Find any service
100+ plugins: Kubernetes, GitHub, PagerDuty, Datadog...
Result: Spotify reports much faster engineer onboarding
Squad Model#
Squad (a small, Scrum-sized team)
- Cross-functional: Engineers, Designer, Product Owner
- Owns: A feature or set of services end-to-end
- Autonomy: Decides how to work (Scrum, Kanban, etc.)
Example: "Search Squad" owns search experience
Tribe (kept under ~100 people)
- Collection of squads in related area
- Sized under the Dunbar number, "smaller than 100 people or so" (Kniberg & Ivarsson, 2012)
Example: "Music Discovery Tribe" includes Search, Browse, Radio
Chapter
- Specialists across squads within a tribe
- Led by Chapter Lead (career growth, standards)
Example: All backend engineers in Music Discovery
Guild
- Community of interest across company
- Voluntary, knowledge sharing
Example: "Web Guild" - all web developers
Service Communication#
# infrastructure/circuit_breaker.py
"""
Circuit Breaker for Service Calls
Protects against cascading failures.
"""
from dataclasses import dataclass
from datetime import datetime
from typing import Callable, Optional, Any
@dataclass
class CircuitBreakerConfig:
failure_threshold: int = 5
success_threshold: int = 3
timeout_seconds: float = 30.0
class CircuitBreaker:
"""
States: CLOSED → OPEN → HALF_OPEN → CLOSED
Applies concepts:
- Week 2: Circuit breaker pattern
- Week 2: Graceful degradation
"""
def __init__(self, name: str, config: CircuitBreakerConfig):
self.name = name
self.config = config
self.state = "CLOSED"
self.failure_count = 0
self.last_failure: Optional[datetime] = None
async def call(
self, func: Callable, *args,
fallback: Optional[Callable] = None, **kwargs
) -> Any:
"""Execute with circuit breaker protection."""
if self.state == "OPEN":
if self._should_try_reset():
self.state = "HALF_OPEN"
elif fallback:
return await fallback(*args, **kwargs)
else:
raise CircuitOpenError(self.name)
try:
result = await func(*args, **kwargs)
self._on_success()
return result
except Exception:
self._on_failure()
if fallback:
return await fallback(*args, **kwargs)
raise
def _on_success(self):
if self.state == "HALF_OPEN":
self.state = "CLOSED"
self.failure_count = 0
def _on_failure(self):
self.failure_count += 1
self.last_failure = datetime.utcnow()
if self.failure_count >= self.config.failure_threshold:
self.state = "OPEN"Deep Dive 4: Event Processing#
"How do you process 500 billion events per day?"
Event Pipeline#
%% title: Spotify event pipeline
flowchart TD
SRC["SOURCES<br/>Mobile/Desktop apps (plays, skips, searches)<br/>Backend services (errors, latency)<br/>External (ad impressions)"]:::client
PS@{ icon: "tech:pubsub", label: "Google Cloud Pub/Sub<br/>1 trillion messages/day capacity<br/>Partitioned by event type<br/>At-least-once delivery" }
SP["STREAMING PATH (Dataflow)<br/>Real-time: Trending now · Live stats"]:::hot
BP@{ icon: "tech:dataproc", label: "BATCH PATH (Dataproc)<br/>Daily/weekly: ML training · Reports" }
subgraph STO["Storage"]
direction TB
BT@{ icon: "tech:bigtable", label: "Bigtable (features)" }
BQ@{ icon: "tech:bigquery", label: "BigQuery (analytics)" }
GCS@{ icon: "tech:gcs", label: "GCS (archive)" }
end
SRC ==> PS
PS ==> SP
PS --> BP
SP ==> STO
BP --> STO
class PS queue
class BP service
class BT,BQ,GCS storeScale: 500B events/day, ~250 TB/day uncompressed (~500 B/event), 10M+ BigQuery queries/month
Implementation#
# data/event_processor.py
"""
Event Processing Pipeline
Processes user events for recommendations and analytics.
"""
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Optional
import json
@dataclass
class StreamEndEvent:
user_id: str
content_id: str
total_played_ms: int
track_duration_ms: int # full length of the track
end_reason: str # "completed", "skipped", "error"
timestamp: datetime
class EventProcessor:
"""
Processes events from Pub/Sub.
Applies concepts:
- Week 3: Stream processing
- Week 8: Late-arriving data
"""
def __init__(self, pubsub, bigtable, bigquery):
self.pubsub = pubsub
self.bigtable = bigtable
self.bigquery = bigquery
self.allowed_lateness = timedelta(hours=24)
async def process(self, raw: bytes) -> None:
event = self._parse(raw)
if not event:
return
# Drop very late events
if datetime.utcnow() - event.timestamp > self.allowed_lateness:
return
# Always write to BigQuery
await self.bigquery.insert('events.streams', event)
# Calculate engagement for recommendations
if isinstance(event, StreamEndEvent):
engagement = self._calculate_engagement(event)
await self.pubsub.publish('taste-updates', {
'user_id': event.user_id,
'content_id': event.content_id,
'engagement': engagement
})
def _calculate_engagement(self, event: StreamEndEvent) -> float:
"""
Engagement scoring:
- Completed: 1.0
- Skipped early: -0.5
- Partial: proportional
"""
if event.end_reason == 'error':
return 0.0
if event.end_reason == 'skipped' and event.total_played_ms < 30000:
return -0.5
if event.track_duration_ms <= 0:
return 0.0
completion = event.total_played_ms / event.track_duration_ms
return min(1.0, completion)Microservices and event processing
Q1/3 The recommendation service starts failing repeatedly. What should a user who taps play experience?
Answer & explanation
Answer: D
The requirement is that recommendations can fail but playback must not. Once the breaker opens, calls go straight to the fallback, so a failing dependency neither blocks nor slows the core feature.
Q2/3 Where should "Trending now" be computed?
Answer & explanation
Answer: B
Trending needs to reflect what's happening now, so it belongs in the streaming path. The batch path suits daily or weekly work like model training and reports, where completeness matters more than latency.
Q3/3 A user skips a track after 12 seconds. What does the event processor publish as the engagement signal?
Answer & explanation
Answer: C
A skip under 30 seconds scores -0.5 and is published to
taste-updates, so recommendations learn what the user dislikes. Only events more than 24 hours late are dropped.
Phase 5: Scaling and Edge Cases#
Scaling for Major Releases#
- Pre-event (1 week before) — Pre-cache album on ALL CDN nodes · Scale API servers 2-3x · Pre-compute recommendation updates · War room ready
- Release moment — Feature flags for instant rollout · CDN serves from cache · Rate limiting protects backends · Async processing for non-critical
- Post-release (first hour) — Monitor error rates · Auto-scale on demand · Graceful degradation if needed
Edge Cases#
1. Cold start (new users)
Solution: Onboarding questions, demographic signals, editorial playlists, rapid learning from first 10 songs
2. Cold start (new songs)
Solution: Content-based + audio analysis, artist fan base, editorial placement
3. Regional licensing
Solution: Rights check at authorization, search excludes unavailable, VPN detection
4. Network transitions
Solution: 20+ second buffer, prefetch next songs, offline cache, graceful quality degradation
Scaling and edge cases
Q1/3 Why pre-cache a huge album on all CDN nodes a week before release?
Answer & explanation
Answer: A
A synchronized release creates a predictable spike. Warming every edge ahead of time means the CDN serves from cache at the release moment, while rate limiting and extra API capacity protect the backend.
Q2/3 A user in a country where a track isn't licensed taps a shared link to it. Where is playback stopped?
Answer & explanation
Answer: C
The rights check runs before any signed URL exists, so no URL means no stream. Search also excludes unavailable tracks, but a shared link skips search, which is why the check has to live in authorization.
Q3/3 A brand-new user has no listening history. Why can't collaborative filtering personalize their first session, and what fills the gap?
Answer & explanation
Answer: B
Collaborative filtering needs a taste vector learned from plays. New users start with explicit signals and curated content, and the system then learns quickly from their first ~10 songs.
Phase 6: Monitoring#
Illustrative targets and readings for this design, not numbers Spotify publishes.
Playback health
| Metric | Target | Current |
|---|---|---|
| Time to first audio (p50) | < 150ms | 142ms |
| Time to first audio (p99) | < 500ms | 380ms |
| Playback success rate | > 99.9% | 99.94% |
| Rebuffer rate | < 0.1% | 0.03% |
CDN health
| Metric | Target | Current |
|---|---|---|
| Cache hit rate | > 95% | 96.2% |
| Edge latency (p99) | < 50ms | 28ms |
API health
| Metric | Target | Current |
|---|---|---|
| Request rate | ~500K/sec | 487K |
| Error rate | < 0.1% | 0.04% |
| Latency (p99) | < 100ms | 67ms |
Data pipeline
| Metric | Target | Current |
|---|---|---|
| Events ingested/sec | ~6M | 5.8M |
| Processing lag | < 5 min | 2.1 min |
Alerts
- CRITICAL: Playback < 99%, API errors > 1%
- WARNING: Cache hit < 90%, Pub/Sub backlog > 10M
Playback health
Q1/3 CDN cache hit rate drops from 96% to 88%. Which playback metric should move first?
Answer & explanation
Answer: B
Every miss adds an origin fetch to that play's start time. The slow tail shows it first; success rate only drops if the origin starts failing too.
Q2/3 Event pipeline processing lag grows from 2 minutes to 40. What do listeners notice right now?
Answer & explanation
Answer: A
Play events leave the client asynchronously and nothing on the play path waits for them. A backlog is a freshness problem for downstream features, not an outage.
Q3/3 Why set targets on p99 time to first audio, not just p50?
Answer & explanation
Answer: B
At hundreds of thousands of plays a second, the slowest 1% is still thousands of people every second. The tail is where the complaints, and usually the real problems, are.
Final Challenge: Count Every Play Exactly Once#
The interviewer turns the marker around. "Last one, and this time it's money."
"Artists are paid from play counts. Your event pipeline is at-least-once, phones play songs offline and upload the events hours later, and some accounts are bots streaming on loop. How do you produce play counts accurate enough to pay royalties on?"
- Events can arrive twice: clients retry, the pipeline redelivers
- Offline plays arrive hours late, sometimes after the day has ended
- Some accounts stream the same tracks around the clock
- Payout reports must be final and auditable
From at-least-once events to exactly-once counts
Where does each problem get solved: duplicates, late plays and fake plays? What does a single play event need to carry for this to work?
One strong answer
- A unique play ID: the client creates it when playback starts, and every retry or redelivery carries the same ID. The pipeline deduplicates on it, so at-least-once delivery becomes exactly-once counting.
- Event time, not arrival time: each event records when the music played. Offline plays are counted into the period they were played in, accepted as late data within an allowed-lateness window before a period's report is finalised.
- A rule for what counts: a minimum listening time before a stream counts (for example, 30 seconds), so skips aren't paid.
- Fraud before payout: per-account and per-device anomaly detection (loops around the clock, many accounts on one device) excludes suspicious plays before reports are finalised.
- Auditable: keep the raw deduplicated events immutable and reconcile reports against them, so any payout can be explained.
Final challenge · Quest check
Q1/4 Events are delivered at-least-once. What turns that into exactly-once play counts?
Answer & explanation
Answer: B
Delivery guarantees stop at the pipeline; correctness comes from an idempotent consumer. The same ID on every copy lets the counter keep exactly one.
Q2/4 A phone plays 30 songs offline on a flight and uploads the events six hours later. How should they be counted?
Answer & explanation
Answer: C
Royalties follow when the music was played. Windows keyed on event time, with allowed lateness and a finalisation cutoff, handle late data without double counting.
Q3/4 Why is (user ID, track ID, upload time) a bad deduplication key?
Answer & explanation
Answer: A
A key that includes the time of sending is new on every retry, the same mistake the payment side quests warn about for idempotency keys. The key has to be fixed when the play happens.
Q4/4 5 billion streams a day is about how many per second on average?
Answer & explanation
Answer: B
5,000,000,000 ÷ 86,400 ≈ 57,900 per second on average, and about three times that at peak.
Interview Conclusion#
"Excellent work. You've demonstrated strong understanding of streaming systems, personalization, and microservices architecture. Any questions?"
"Thank you! How does Spotify prevent filter bubbles in recommendations?"
"Great question. Spotify hasn't published its exact mix, but in your design the 20% exploration ratio is the key lever, and Spotify's own features like 'Enhance' add variety on top."
Summary: Concepts Applied#
- Week 1: Data at scale
- Partitioning — User data by
user_id, events by timestamp - Replication — Multi-region audio and user data
- Hot keys — Popular songs cached at all CDN edges
- Week 2: Failure-first design
- Circuit breakers — Protect recommendation service
- Timeouts — 30ms budget for authorization
- Graceful degradation — Popular content fallback
- Week 3: Messaging & async
- Pub/Sub — 500B+ events/day
- Async processing — Recommendation updates
- Week 4: Caching
- Multi-tier — Client → CDN → Origin
- Hot content — Always cached, never evicted
- Week 5: Consistency
- Eventual — Recommendations, play counts
- Strong — Playlists, user library
- Week 8: Analytics pipeline
- Stream — Dataflow for real-time
- Batch — Dataproc for ML training
- Week 10: Production readiness
- SLOs — < 200ms playback, 99.99% availability
- Observability — Comprehensive metrics
Scale
- 777 million users across 184 markets
- 1+ trillion events collected daily (2024)
- Near-instant starts: a 265 ms median was already reported back in 2010
Personalization
- ~1 in 3 new artist discoveries via personalized recommendations
- Discover Weekly: 100B+ tracks streamed since 2015
- Three-model hybrid approach
Engineering culture
- Squad model influenced entire industry
- Backstage: Open-sourced developer portal
- 2,000+ microservices, autonomous teams
Sources#
Official:
- Blog
Spotify Engineering Blog: https://engineering.atspotify.com/
- Read
Backstage: https://backstage.io/
- Read
Spotify for Developers: https://developer.spotify.com/
Statistics:
- Read
Spotify company info: https://newsroom.spotify.com/company-info/
- Read
Spotify Q2 2026 results (Form 6-K): https://www.sec.gov/Archives/edgar/data/0001639920/000114036126031044/ef20078867_ex99-1.htm
- Read
Spotify Loud & Clear (royalties): https://loudandclear.byspotify.com/
- Read
Discover Weekly turns 10: https://newsroom.spotify.com/2025-06-30/discover-weekly-turns-10-celebrating-100-billion-tracks-streamed-and-a-decade-of-personalized-discovery/
- Blog
Data Platform Explained Part II (1T+ events/day): https://engineering.atspotify.com/2024/5/data-platform-explained-part-ii
- Read
Kreitz & Niemelä, "Spotify – Large Scale, Low Latency, P2P Music-on-Demand Streaming" (IEEE P2P 2010)
- Read
Backlinko - Spotify Statistics 2025: https://backlinko.com/spotify-users
- Read
DemandSage - Spotify Stats: https://www.demandsage.com/spotify-stats/
Architecture:
- Read
Google Cloud - Spotify Case Study: https://cloud.google.com/customers/spotify
- Blog
Spotify CDN: https://engineering.atspotify.com/2020/02/how-spotify-aligned-cdn-services-for-a-lightning-fast-streaming-experience
Organization:
- Paper
Spotify Squad Model: https://blog.crisp.se/wp-content/uploads/2012/11/SpotifyScaling.pdf
Self-Assessment Checklist#
After studying this case study, you should be able to:
- Design multi-tier caching for media delivery
- Explain collaborative filtering and matrix factorization
- Design hybrid recommendation systems
- Handle cold start for users and content
- Architect event pipelines at 500B+ events/day
- Implement circuit breakers and graceful degradation
- Design URL signing for CDN authentication
- Explain streaming vs batch processing trade-offs
- Design for predictable and unpredictable traffic spikes
- Articulate the squad model for team organization
- Design developer portals for microservices
This case study covers Spotify's architecture for delivering personalized audio experiences at massive scale.
this lesson in shorts · 3
play them all →One idea from this lesson in about a minute each, made for your phone.
- 0/7stages cleared
Follow-up drills
0 of 11 practisedYou presented Spotify. Now the interviewer pushes back. Answer each follow-up out loud or in the box, then reveal a strong answer.
“Your estimate puts peak egress at 5.5 Tbps. Is bandwidth the hard part of this design?”
Pushes on · knowing which number actually drives the design
Go deeper · reading trail
all reading →Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.
Spotify's Event Delivery – The Road to the Cloud (Part I)
Igor Maravić · 2016- What
- The first post in Spotify's series on rebuilding the pipeline that collects client events.
- Key idea
- Every play and search becomes an event; the old Kafka and Hadoop pipeline is explained, and why Spotify chose Google Cloud managed services for the new one.
- Why here
- The event processing deep dive of this quest: play events feed royalties and recommendations, so delivery must be complete.
Personalization at Spotify using Cassandra
Kinshuk Mishra, Matt Brown · 2015- What
- How Spotify's personalisation stack served recommendations for 60 million users.
- Key idea
- Kafka collects logs, Storm processes them in real time and Crunch runs batch jobs on Hadoop, and Cassandra serves the resulting user profiles at low latency.
- Why here
- The hybrid recommendation architecture of this quest: batch and streaming models meeting in one serving store.
What the Heck is Backstage Anyway?
2020- What
- Spotify explains Backstage, its developer portal, the day after open-sourcing it.
- Key idea
- A single portal catalogues every service, its owners and docs, with plugins for tooling, so thousands of engineers can find and run things in one place.
- Why here
- The 'Microservices & Backstage' deep dive of this quest, from the team that built it.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.