Re-ranking: Scoring Code, Production Trade-offs, and a Transparent Research Case Study

A Simplified Cross-Encoder-Style Scorer

Real cross-encoders (like the BERT-based model from Nogueira & Cho’s paper) learn this joint scoring through training on millions of examples. This simplified, hand-written version illustrates the same core idea — reading the query and document together, rather than comparing independent vectors — without needing an actual model:

import re

def tokenize(text):
    return set(re.findall(r'[a-z0-9]+', text.lower()))

def cross_encoder_score(query, document):
    q_tokens = tokenize(query)
    d_tokens = tokenize(document)
    overlap = q_tokens & d_tokens

    # joint signal: does the document contain an action word that
    # matches an action-seeking query? A bi-encoder comparing two
    # independent vectors has no direct way to check this interaction —
    # a cross-encoder can, because it reads both at once.
    action_words = {'replace', 'install', 'remove', 'insert', 'unscrew', 'steps'}
    has_action_match = (
        bool(d_tokens & action_words) and bool(q_tokens & {'replace', 'how'})
    )

    base = len(overlap) / max(len(q_tokens), 1)
    bonus = 0.5 if has_action_match else 0.0
    return round(min(base + bonus, 1.0), 2)

Running this against the three chunks from the Introduction level produces scores of 0.3, 1.0, and 0.1 for chunk_A, chunk_B, and chunk_C respectively — correctly identifying chunk_B (the actual replacement steps) as the strongest match, the same result shown earlier.

This is a hand-written illustration of the concept, not a real trained model — labeled as such deliberately, consistent with how every worked example in this series is presented.

Measuring It With precision@k

Extending the scenario to five candidates, with two of them (chunk_B and chunk_D) marked as the actual ground-truth relevant chunks, shows the improvement in concrete numbers:

Before re-ranking, precision@2 is 0.00 — neither of the top two results is actually relevant. After re-ranking, precision@2 is 1.00 — both truly relevant chunks now occupy the top two positions. This is the same real/false-positive pattern covered in RAG Advanced’s precision/recall code, applied specifically to the re-ranking step.

When Re-ranking Isn’t Worth Adding

Re-ranking is not a universal upgrade — it’s a trade of latency for accuracy, and that trade doesn’t make sense everywhere:

• Latency budget too tight

  A cross-encoder pass adds real time per query. If a use case needs sub-100ms responses, adding a re-ranking step may not fit at all.

• Initial retrieval is already accurate

  If bi-encoder retrieval already puts the right chunk first for a given domain, a second pass adds cost without meaningfully improving the outcome.

• Very high query volume

  Cross-encoder inference is more expensive per query than a bi-encoder similarity lookup. At large scale, that cost multiplies fast — worth measuring before adding it everywhere.

The latency this adds compounds with everything else in the pipeline — worth weighing against the full breakdown in RAG Pipeline Architecture Intermediate, where generation already dominates total response time.

Local Reranking: Running a Cross-Encoder with Ollama and Python

Every other model covered in this series — Ollama for generation, embedding models for retrieval — can be self-hosted with no cloud bill. It’s natural to expect re-ranking to work the same way. It mostly does, but with a real gap worth knowing about.

BGE-reranker models are genuinely available to pull through Ollama’s registry — ollama pull qllama/bge-reranker-v2-m3 works. But Ollama has no dedicated reranking endpoint: as of this writing, that model can only be called through Ollama’s /api/embed endpoint, which returns an embedding vector — not the joint query-document relevance score a cross-encoder is actually supposed to produce. Native reranking support has been an open feature request on Ollama’s GitHub since 2024, still unresolved. Pulling a reranker model through Ollama today doesn’t get you real cross-encoder scoring, whatever the model’s name suggests.

Source: ollama/ollama, Issue #3368, “Reranking models”

Real local cross-encoder scoring is still entirely possible — just through a Python library instead, running the actual model architecture a reranker needs:

from sentence_transformers import CrossEncoder

model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

query = 'How do I replace the battery in the XR-400 sensor?'
docs = [
    'XR-400 battery specifications: CR2032, 3V, approx. 18 month life under normal usage.',
    'To replace the XR-400 battery: unscrew the rear panel, lift out the old CR2032, insert the new one matching polarity, and re-seal the panel.',
    'Battery disposal regulations vary by region; check local e-waste guidelines before discarding CR2032 cells.',
]

pairs = [[query, doc] for doc in docs]
scores = model.predict(pairs)

This is a real, trained cross-encoder — not the simplified scorer from earlier on this page — run locally and produced these actual scores:

-0.4628  XR-400 battery specifications: CR2032, 3V, approx. 18 month...
 9.0420  To replace the XR-400 battery: unscrew the rear panel, lift...
-10.6192 Battery disposal regulations vary by region; check local e-w...

Cross-encoder scores aren’t probabilities and aren’t bounded between 0 and 1 — they’re raw relevance logits, useful only for ranking relative to each other within one query. Here the gap is stark: the chunk with the actual replacement steps scores +9.04, while the two irrelevant-but-keyword-adjacent chunks score deeply negative. A real trained model confirms the same ranking the hand-written scorer produced earlier, with a far wider margin.

Case Study: How Claude Re-ranks Its Own Research

This isn’t a claim that Claude is “the best” AI model — that’s not a verifiable statement, and it’s not what this section argues. It’s a specific, checkable account of the actual two-stage process Claude followed to research this exact page, because that process really does mirror the retrieve-then-re-rank pattern this article is about.

Stage 1: Broad recall, not precision

Writing the “Real Re-ranker Models” section on the Intermediate level started with a broad web search for current reranker models — Cohere Rerank, BGE variants, MS MARCO cross-encoders. That search returned a wide, messy set of results: blog posts, comparison articles, vendor pages, GitHub issues. Nothing in that first pass was trusted as fact yet — it was purely a candidate-generation step, exactly like a bi-encoder pulling back 50 approximate matches without claiming any one of them is the answer.

Stage 2: Reading each candidate before trusting it

The claim above this section — that Ollama has no native rerank endpoint and BGE models only work through /api/embed — is a good concrete example. The initial search surfaced that claim indirectly, as a summary. It was not published on that basis. Claude fetched the actual GitHub issue directly, found it was an open, unresolved feature request with no maintainer confirmation either way, and then fetched a real Ollama model page for a pulled BGE-reranker model to check what usage instructions it actually gave. Only after seeing, on the real page, that the only documented usage was the /api/embedcall — with no mention of reranking at all — did that claim get treated as solid enough to publish. Most of what the first search surfaced never made it into this article at all, the same way most of a bi-encoder’s 50 candidates never reach the final answer.

The clearest example: code that had to actually run

The local cross-encoder code just above wasn’t written from memory and left untested. Claude ran it for real, on this machine, which downloaded the actual cross-encoder/ms-marco-MiniLM-L-6-v2 model and produced the exact scores shown (-0.4628, 9.0420, -10.6192) — not illustrative numbers chosen to make a point, but the literal output of a real model run once and copied in directly. That’s the same discipline applied to every runnable snippet across this entire site: a claim about what code produces doesn’t get published until the code has actually produced it, checked once in Python and once in Node wherever both matter.

The honest limit of this comparison: there’s no trained scoring model deciding which sources to keep — it’s a deliberate, repeatable habit of fetching primary sources and running code before citing either. But the shape holds: cheap, wide recall first, then a slower pass that only keeps what survives direct verification. The rest gets discarded, the same way a re-ranker discards the candidates that don’t hold up once actually examined.

Fun Fact

The same two-stage shape — cheap broad step, then a slower careful one — shows up all over information systems, not just RAG: search engines rank billions of pages with fast signals first, then apply more expensive relevance models only to the shortlist that survives.

Test Yourself