RAG in Production: Precision/Recall, Grounding & Faithfulness

Precision and Recall, in Code

A small worked example: a 10-chunk knowledge base where only 3 chunks are actually relevant to a given query, and a retriever that returns its top 5 candidates.

relevant_chunks = {"chunk_2", "chunk_5", "chunk_9"}  # ground truth
retrieved_chunks = ["chunk_2", "chunk_3", "chunk_5", "chunk_7", "chunk_8"]  # top-5 retrieved

def precision_at_k(retrieved, relevant):
    hits = [c for c in retrieved if c in relevant]
    return len(hits) / len(retrieved), hits

def recall_at_k(retrieved, relevant):
    hits = [c for c in retrieved if c in relevant]
    return len(hits) / len(relevant), hits

precision, hits = precision_at_k(retrieved_chunks, relevant_chunks)
recall, _ = recall_at_k(retrieved_chunks, relevant_chunks)

print(f"Precision@5: {precision:.2f} ({len(hits)}/5)")
# Precision@5: 0.40 (2/5)
print(f"Recall@5: {recall:.2f} ({len(hits)}/3)")
# Recall@5: 0.67 (2/3)
print("Missed relevant chunk:", relevant_chunks - set(retrieved_chunks))
# Missed relevant chunk: {'chunk_9'}

chunk_9 is genuinely relevant, but it never made it into the top 5 — the model generating the answer has no way to use information it was never given, regardless of how well it reads the four chunks it did receive. Precision and recall often trade off against each other: try the three retriever qualities below and see how the missed chunks change.

Grounding: RAG Can Still Hallucinate

Perfect retrieval doesn’t guarantee a correct answer. A model can still misread a retrieved chunk, mix details from two different chunks into one incorrect claim, or answer using only the part of a chunk it “noticed” while ignoring a contradicting detail elsewhere in the same text. Grounding techniques push back against this directly: prompting the model to cite exactly which retrieved chunk supports each claim it makes, so a reader — or an automated check — can verify the answer against the actual source rather than trusting it by default.

Here’s exactly that failure happening: two chunks are retrieved correctly — one about the XR-400 sensor, one about the unrelated XR-500 — but without grounding, the model blends a spec from the wrong chunk into its answer:

Both chunks were genuinely relevant to a query about sensor specs in general, and retrieval did its job correctly. The failure happens entirely during generation — the model has two similar-looking chunks in its context and picks up the wrong one’s detail. Grounding fixes this not by improving retrieval, but by forcing the model to justify each claim against a specific source, which makes a blending error like this one immediately visible instead of silently wrong.

Evaluating RAG: Faithfulness

RAGAS, a widely used open-source evaluation framework built specifically for RAG pipelines, formalizes this gap as faithfulness— evaluating a model’s ability to use retrieved passages faithfully, rather than just checking whether retrieval found the right chunks in the first place. It’s deliberately measured separately from retrieval quality: a system can score well on recall while still scoring poorly on faithfulness, if the model keeps misusing the context it’s actually given. This mirrors the same evaluation discipline covered in Comparing Ollama Models Advanced: measure the specific thing you actually care about, not a proxy that merely correlates with it.

Source: Es et al., “Ragas: Automated Evaluation of Retrieval Augmented Generation,” arXiv:2309.15217

A Practical Guideline for Checking Faithfulness

1. 1. Break the answer into individual claims

   Split the generated answer into separate, atomic factual statements — not paragraphs, single claims. “18-month battery life” and “IP67 rating” are two different claims, even if the answer states them in one sentence. This is the same breakdown RAGAS itself performs before scoring anything.

2. 2. Check each claim against the retrieved context — not general knowledge

   For every claim, ask only one question: does the actual retrieved text support this, word for word or in clear substance? A claim can be true in the real world and still fail this check, if the context handed to the model never actually said it. Outside knowledge doesn’t count — faithfulness measures whether the model stuck to what it was given, not whether it happened to be right.

3. 3. Count supported claims over total claims

   Faithfulness score = (number of claims supported by context) ÷ (total number of claims in the answer). A single unsupported claim in an otherwise accurate answer still lowers the score — faithfulness doesn’t average out a good answer against one invented detail, it counts it as a real failure.

4. 4. Trace every unsupported claim back to where it likely came from

   An unsupported claim is rarely random — it’s usually the model filling a gap with a plausible-sounding guess, or pulling in a detail from a different chunk (the blending failure covered above). Identifying which is which tells you whether the fix is a better prompt, better grounding, or better retrieval.

Try the Guideline Yourself

Apply steps 1–3 to a real answer: decide whether each claim below actually holds up against the retrieved context.

Fun Fact

RAGAS was specifically designed to work without ground-truth human answers — it evaluates faithfulness, relevance, and retrieval quality using other models as judges, the same LLM-as-judge idea covered in Comparing Ollama Models Advanced, which is exactly what makes it practical to run continuously in production rather than only during a one-off manual evaluation.

Test Yourself