Day 1Search Fundamentals & Architecture

Why LIKE fails. LIKE '%red%' can't use a B-tree, which is sorted by string start, so 50M rows get a full scan. It also has no relevance ranking, typo tolerance, word boundaries, stemming or synonyms.

The inverted index. Pre-compute word to documents instead of scanning documents for words. A query is one lookup per term plus an intersection, smallest list first: "running shoes" takes under a millisecond.

Text analysis. Character filters (strip HTML), then the tokenizer, then token filters (lowercase, stop words, stemming, synonyms). The same analyzer must run at index time and query time, or "Running" never matches the indexed "run".

The multi-field pattern. One source field, several representations: brand as text for search, brand.keyword for facets and sorting, brand.autocomplete for typeahead. IDs are keyword: they're only exact-matched.

Scatter-gather. A coordinating node sends the query to every shard (primary or replica). Each searches its own inverted index and returns its top N with scores; the coordinator merges, sorts by score and returns the overall top N.

The product search cluster. 20 primary shards plus 20 replicas on 6 data nodes, 3 dedicated master nodes, 2 coordinating nodes. Text goes in must to drive the score; category, price and stock go in filter.

• Different analyzers at index and query time, so tokens never match.
• Oversharding: per-shard memory and file-handle overhead degrades queries.
• Not planning primary shards: the count is fixed at creation, so changing it needs _split/_shrink or a reindex.

Day 2Indexing Pipeline

Never dual-write. Writing PostgreSQL then Elasticsearch breaks four ways: the ES write fails, the process crashes between writes, a partition holds the transaction open, or retries reorder updates so ES ends at $100 while PostgreSQL has $120.

Change data capture. App to PostgreSQL; Debezium reads the WAL through logical replication and publishes change events to Kafka; the indexing pipeline consumes them into Elasticsearch. The app writes to one place, and events replay within Kafka's retention.

Version by WAL position. Each bulk operation carries version: event.lsn with version_type: external. If an older change arrives late after retries or reordered batches, ES rejects it with a version conflict instead of overwriting newer data.

Batch, but by priority. Flush at 500 documents or after 5 seconds, so quiet periods don't strand updates. Stock going to zero is CRITICAL in 10-document batches; description edits are NORMAL at 500; bulk imports LOW at 1,000.

Zero-downtime reindex. Reads stay on the products alias over v1 while updates go to both indexes and old data is copied with op_type: create. Verify counts, switch the alias in one atomic remove+add, then delete v1.

• Running a big import through the live pipeline, starving CDC updates.
• Skipping index aliases, so a reindex can't switch with zero downtime.
• Blaming the pipeline when the ES document is right but a result cache is stale.

Day 3Query Processing & Relevance

BM25 in one line. Score grows with term frequency but saturates (k1, default 1.2), rare terms weigh more (IDF), and long documents are normalized (b, default 0.75). 10 occurrences aren't 10x better than 1.

Rare terms win. In the example corpus, "nike" (2% of documents) has IDF 3.912, "shoes" (10%) 2.3026, and "the" (90%) 0.1054. A match on the rarer term says more about relevance.

Query context vs filter context. What the user typed is scored in query context. What they selected (facets, toggles, ranges) goes in filter: yes or no, unscored, cacheable as a bitset. The lesson puts the split at ~3x faster.

Boost in layers. Start with field boosts (name^3, brand^2, description), add business boosts in should clauses, and use function_score (popularity, recency decay, rating, featured) only when needed. A/B test every ranking change.

Never return nothing. Progressive relaxation: exact AND query, then drop low-priority filters (rating, price), then fuzzy matching, then any-term matching, then no filters at all. Category stays: it's a strong intent signal.

Pick the multi_match type. best_fields with tie_breaker 0.3 is the product default. cross_fields lets "John" match first_name and "Smith" last_name. most_fields sums scores across analyses. phrase_prefix and bool_prefix serve autocomplete.

• Putting category, brand and price in must: they sway scores without adding relevance and aren't cached.
• Filtering out-of-stock items away when they should be demoted with a boosting query.
• Over-boosting, which makes relevance unpredictable.

Day 4Typeahead at Scale: Autocomplete, Spelling and Facets

Typeahead multiplies traffic. One request per keystroke: a search is ~12 requests. The store's ~2,800 searches a second at peak become ~33,000 typeahead requests, so typeahead is its own cheap system, not prefix queries on the search cluster.

Store the answer, not the means. A plain trie ranks whole subtrees, largest for short prefixes. Precompute each prefix's top-k at build time: serving is one walk or map lookup, O(prefix length). An FST is the compressed, immutable version.

Suggestions come from query logs. Normalize, count searches and distinct users, decay with a half-life (14 days, illustrative), filter, then ship a versioned snapshot nightly. A stream job adds a small trending overlay, capped in slots and filtered more strictly.

Suggestions are the system speaking. A distinct-user minimum keeps one person's phone number out. PII patterns, blocklists and policy lists drop the rest, including unconfirmed rumours. A serve-time blocklist plus CDN purge removes misses in seconds.

Global at the edge, history private. The global list is the same for everyone per prefix and locale, so the CDN caches it on that key. History comes privately, merged on the client. Never cache a blended list.

After Enter. Spelling: pick the c maximizing P(c) x P(w | c), with your query logs as the language model. Facets: selections in post_filter, each facet filtered by every selection except its own, so Adidas stays countable.

• A prefix query on the search cluster per keystroke: a sale night slows search for everyone.
• Raw all-time counts: old models win forever and one user's query leaks.
• Sharding by first letter: skew, and launches make one shard hot.

Day 5Location Search and Search at Scale

Why a B-tree can't do "near". An index on (lat, lng) uses only the latitude range, then tests longitude on every row: a strip across the country. Instead, cover the query with cells, then refine by exact distance.

Geohash and its edge. Each base-32 character adds 5 bits and divides the cell by 32. A shared prefix means close, not the reverse: points 43 m apart can share only tdr. Search the cell plus 8 neighbours, then refine.

S2 and H3. S2 numbers a quadtree on a cube in Hilbert order: a cell's descendants are one ID range, ideal for coverings and shard keys. H3 hexagons have six equidistant neighbours, so rings approximate circles; children don't nest exactly.

Shapes in a database. PostGIS puts an R-tree on GiST that stores bounding boxes; ST_DWithin uses it, ST_Distance in WHERE doesn't. Delivery zones are polygons: ask which zones contain the customer's point.

Moving objects are soft state. Overwrite each partner's position in memory on every ping; touch the cell index only on a cell change. A TTL drops dead phones. The trail goes to Kafka, never into the search index.

Stop kNN with a bound. Grow H3 rings outward, but stop only when you have k results and the k-th is no farther than a lower bound on anything unscanned. Re-rank the top few by road ETA.

• Updating a search document on every ping: reindexing and segment churn slow product search.
• Trusting the prefix: missing a partner 43 m away across a cell edge.
• One shard per city: one stadium slows the whole city.

BossDesigning a Legal Document Search System

Size the corpus. 100M documents (50M public, 50M private) at ~65 KB is 6.5 TB of text. 2x index overhead and 2x for replicas make 26 TB in Elasticsearch, plus 50 TB of original PDFs in S3: ~76 TB.

Size the traffic. 50,000 concurrent users at 20 searches an hour is 1M an hour, ~280 a second; autocomplete at 5 keystrokes per search is ~1,400 a second. Today's 10,000 a minute (~170/s) leaves ~1.7x headroom.

A real query parser. A lexer and parser turn (breach /5 contract) AND damages NOT punitive into must, should and must_not clauses. Proximity becomes span_near with in_order: false; /s means ~15 words, /p ~50.

Isolation in layers. Per-tenant indices, a service-injected tenant_id filter, document-level security, a tenant taken from the validated JWT (never the request body), and audit logs. Zero tolerance means no single control is trusted alone.

Citations as edges. A separate index of {citing_id, cited_id, treatment} makes "who cites X" a term query on cited_id, with aggregations for treatment breakdowns, "good law" checks and 2-hop chains.

Triage every upload. Store the original in S3 first. A PDF with under 100 characters of text in its first pages is scanned: render each page at 300 DPI and OCR it. Word goes through Tika. Then publish to Kafka.

• Trusting one isolation control, or a tenant_id from the request body.
• Using a phrase query for proximity: "liability for negligence" should match negligence /10 liability.
• Fuzzy matching citation variants instead of a canonical case ID plus name synonyms.