How you represent documents determines what you can find

The first lesson was about what happens before a query touches the index. This one is about the index itself — specifically, what lives inside it.

The architecture of your retrieval system determines what kinds of matches are even possible. Dense retrieval can find synonyms that BM25 would miss entirely. ColBERT can catch subtle token-level relevance signals that a single vector would average away. Cross-encoders can rank with precision that first-stage systems can only approximate.

There are four major families, each with a different answer to the same question: how should a document be represented so the right one gets found?

One vector, one document

Dense retrieval is the default choice for semantic search. The mechanism is straightforward: you encode both the query and each document into a single dense vector using the same model. At query time, you search for the nearest vectors in that high-dimensional space using approximate nearest neighbor search.

The power is in what the vector captures. Dense models learn to encode meaning, not just tokens — so "automobile" and "car" end up close together, and a query about one retrieves documents about the other. This vocabulary mismatch handling is what BM25 fundamentally cannot do.

But the same property is also the weakness. Exact, rare strings — a CVE number, a specific product SKU, a precise error code — may not match reliably if they appeared infrequently during training. The model simply didn't see enough examples to learn that "CVE-2024-1234" means something specific. This is why dense retrieval almost never ships alone in production.

BM25, but learned end-to-end

Dense retrieval replaced BM25 by throwing it out. SPLADE replaced it by learning to do it better.

SPLADE trains a model to produce sparse vectors — vectors where most dimensions are zero, but the non-zero dimensions correspond to vocabulary tokens with learned weights. The structure looks exactly like a BM25 representation, but those weights came from end-to-end training on retrieval tasks, not from term frequency statistics.

The key property is implicit query expansion. When you search for "car," SPLADE may also activate weights for "vehicle," "automobile," and "sedan" — because it learned from training data that documents using those terms are relevant to "car" queries. You get vocabulary expansion for free, without building or maintaining a synonym dictionary.

In practice, SPLADE beats BM25 significantly on BEIR benchmarks while remaining indexable with a standard inverted index. That matters architecturally: if you're already running Elasticsearch or OpenSearch, SPLADE is a drop-in upgrade path to better recall without adding vector database infrastructure.

Keep every token, score every match

Dense retrieval's weakness is compression. Collapsing all the semantic information in a 512-token document into a single vector means throwing most of it away. ColBERT goes the other direction entirely.

ColBERT keeps every token embedding, for every document. Where dense retrieval produces one vector per document, ColBERT produces one vector per token. At query time, it scores relevance using MaxSim: for each query token, find its best-matching document token and record that similarity. Sum those per-token max similarities across all query tokens to get the final document score.

The result is token-level expressiveness without the prohibitive cost of cross-encoders. ColBERT v2 achieves near cross-encoder quality at speeds much closer to bi-encoder retrieval, using compressed token embeddings that bring storage requirements back toward earth. For legal, medical, or code search — where retrieval quality matters more than infrastructure cost — it's often the highest-quality single-stage option available.

Read both texts together

Everything we've covered so far — dense retrieval, SPLADE, ColBERT — encodes queries and documents independently. Cross-encoders do something categorically different: they read the query and document together, concatenated, and score relevance from that joint representation.

That joint reading is what makes cross-encoders powerful. Bi-encoders can miss cross-text interaction — whether a document passage is relevant might depend on context established in the query that a single vector can't fully capture. Cross-encoders see both texts completely and score accordingly. The quality gap over bi-encoders is consistent and meaningful.

But joint reading is also what makes cross-encoders expensive — you can't pre-compute anything for a new query because the model needs both texts at once. So cross-encoders are almost always a second stage: retrieve a top-K candidate set quickly with a fast first-stage system, then rerank those K candidates with the cross-encoder before returning the top 10 to the application.

Build in two stages

The two-stage pipeline is the production default for a reason. Stage one is fast retrieval for recall — dense ANN search, hybrid dense+sparse with RRF, or SPLADE if you want the inverted index. Stage two is a cross-encoder for precision — take your top 100 candidates from stage one and rerank them before returning the top 10.

Each stage improves independently. You can swap embedding models in stage one without touching the reranker. You can upgrade the reranker without rebuilding the index. And because each stage has a clear optimization target — recall for stage one, precision for stage two — you always know which knob to turn when results fall short.

ColBERT can replace either stage or both, at higher storage cost. It earns its place when you need maximum retrieval quality and can afford the infrastructure — legal and medical search are the canonical use cases.

In the next lesson, we'll look at fusion — how to combine results from multiple retrieval passes into a single ranked list that's better than either pass alone.