What is re-ranking in a RAG pipeline?

Re-ranking is a second retrieval step that takes a larger set of initially retrieved candidates — say, the top 50 — and re-scores them with a slower but more accurate model that actually reads the query and each candidate together. It then reorders the candidates so the most genuinely relevant ones end up first, before only the top few are handed to the model that generates the final answer.

AI & ML — INTRODUCTION

Re-ranking: The Second Pass That Fixes Retrieval Mistakes

The first search a RAG pipeline runs is fast — and often wrong about the order. Re-ranking is the step that catches that mistake before it reaches the model.

The Quick Answer

Retrieval in a real RAG system usually happens in two stages. The first stage — a vector database search, keyword search, or both — is fast, but only approximately accurate: it pulls back a fairly wide set of candidates (say, the top 50) rather than trying to get the final order exactly right in one pass. Re-ranking is the second stage: a slower, more careful model reads the actual query together with each candidate and re-scores how relevant it really is, reordering the set before only the top few are handed to the model that generates the answer.

Why the First Pass Gets It Wrong

Fast retrieval methods judge relevance using shortcuts — how many words overlap, or how close two pieces of text sit in embedding space. Those shortcuts are usually good enough to find a reasonable set of candidates quickly, across a huge collection of documents. But “usually good enough” isn’t the same as correct: a chunk can share a lot of surface vocabulary with a query without actually answering it, and a chunk that genuinely answers the question can rank lower simply because it phrases things differently.

See It Happen

Here are three candidate chunks retrieved for the same question. Toggle between the initial retrieval order and the order after re-ranking.

The initial pass ranks the chunk that actually explains how to replace the battery last, because it repeats the words “battery” and “XR-400” less often than the other two. Re-ranking reads all three chunks against the actual question and correctly promotes it to first.

What the Experts Say

A well-known 2019 paper by Nogueira & Cho showed just how large this gap can be. Using a transformer model that reads the query and passage together — the same idea shown above — their re-ranker topped the MS MARCO passage retrieval leaderboard, improving the key ranking metric (MRR@10) by 27% relative to the previous best system.

Source: Nogueira & Cho, “Passage Re-ranking with BERT,” arXiv:1901.04085

Fun Fact

Re-ranking was already mentioned briefly back in RAG Intermediate as one part of a bigger pipeline. This series takes it further: the actual difference between the two kinds of models involved, real re-ranker models used in production, and the production trade-offs of adding this extra step.

Test Yourself

Why is retrieval usually done in two steps instead of one?

In the before/after example on this page, why did the first retrieval step rank the chunk with the actual answer last?

What did Nogueira & Cho (2019) demonstrate about using BERT for re-ranking?

Ready to see the actual difference between the two kinds of models involved, and real re-ranker models used in production? Continue to the Intermediate level →