What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder embeds a query and a document separately, then compares the two resulting vectors with a fast similarity calculation — this is what makes vector search over millions of documents practical. A cross-encoder feeds the query and document into the model together, letting it directly attend to both at once, which produces a far more accurate relevance judgment but is too slow to run over an entire collection. Re-ranking uses a cross-encoder, but only on the small candidate set a bi-encoder already narrowed down.

← Back to the Introduction level

AI & ML — INTERMEDIATE

Re-ranking: Bi-Encoders vs Cross-Encoders, and How to Measure the Gain

The Introduction level showed re-ranking fix a bad ordering, but glossed over why the re-ranking model can afford to be so much more careful than the retrieval model that came before it. The answer is architectural, not just a difference in effort.

The Quick Answer

Initial retrieval uses a bi-encoder: it embeds the query and every document separately, so all the document embeddings can be computed once, in advance, and stored. A query only needs to be embedded once per search, then compared against those pre-computed vectors with fast math. Re-ranking uses a cross-encoder: it feeds the query and a candidate document into the model together, letting it directly weigh how the two relate to each other — far more accurate, but far too slow to run over an entire collection. That’s why re-ranking only ever runs on the small candidate set a bi-encoder already narrowed down.

Why the Cost Difference Is So Large

A bi-encoder’s core trick is doing the expensive part — running the model — only once per item, ever. The same embedding for a document can be reused for every future query against it. A cross-encoder has no such shortcut: every new query needs to be paired with every candidate document all over again, because the model’s judgment depends on seeing both together. The paper that introduced this bi-encoder approach for sentence embeddings (Sentence-BERT) measured the practical difference directly: finding the most similar pair among 10,000 sentences took about 65 hours with a BERT-style cross-encoder, versus about 5 seconds with the bi-encoder approach — while keeping essentially the same accuracy for that comparison task.

Source: Reimers & Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” arXiv:1908.10084

This is exactly why a two-stage pipeline exists at all: a bi-encoder’s speed makes it the only realistic option for searching a huge collection (the same approach covered in Embeddings Intermediate), while a cross-encoder’s accuracy makes it worth the extra cost — but only once the candidate set is already small.

Real Re-ranker Models

Production systems don’t need to train a cross-encoder from scratch — dedicated re-ranker models are available as both APIs and self-hosted open-weight models:

Cohere Rerank — a hosted API specifically built for re-ranking, with no infrastructure to manage.
BGE-reranker (BAAI) — an open-source cross-encoder family, including a multilingual variant, that can be self-hosted like any other model covered in Ollama.
cross-encoder/ms-marco models— smaller open cross-encoders trained on the MS MARCO dataset, often used for prototyping since they’re free and light enough to run on CPU.

Measuring the Improvement: nDCG

“The right chunk was somewhere in the results” isn’t precise enough to tell whether re-ranking actually helped — position matters too. nDCG (Normalized Discounted Cumulative Gain) scores a ranked list by rewarding relevant results more when they appear near the top, and normalizes that score against the best possible ordering of the same items, giving a score between 0 (worst) and 1 (perfect):

DCG = Σ (relevance_i / log₂(position_i + 1))
nDCG = DCG / IDCG (the DCG of the ideal ordering)

Grading the same three chunks from the Introduction example on a 0–2 relevance scale (0 = irrelevant, 2 = directly answers the query) shows the gap in numbers:

Fun Fact

Because a cross-encoder has to process every query-document pair fresh, re-ranking adds real latency to every single request — a cost worth keeping in mind alongside the latency breakdown covered in RAG Pipeline Architecture Intermediate.

Test Yourself

Why can’t a cross-encoder be used for the initial retrieval step over an entire document collection?

What does nDCG measure that a simple “did we retrieve the right chunk” check does not?

According to Reimers & Gurevych (2019), what was the practical computational cost of finding the most similar pair among 10,000 sentences using a cross-encoder-style BERT approach, versus a bi-encoder (SBERT) approach?

Ready for the real scoring math behind a cross-encoder, when to skip re-ranking entirely, and a look at how this article’s own research process mirrors the same two-stage pattern? Continue to the Advanced level →