RAG in Production: Precision/Recall, Grounding & Faithfulness

Precision and Recall, in Code

A small worked example: a 10-chunk knowledge base where only 3 chunks are actually relevant to a given query, and a retriever that returns its top 5 candidates.

relevant_chunks = {"chunk_2", "chunk_5", "chunk_9"}  # ground truth
retrieved_chunks = ["chunk_2", "chunk_3", "chunk_5", "chunk_7", "chunk_8"]  # top-5 retrieved

def precision_at_k(retrieved, relevant):
    hits = [c for c in retrieved if c in relevant]
    return len(hits) / len(retrieved), hits

def recall_at_k(retrieved, relevant):
    hits = [c for c in retrieved if c in relevant]
    return len(hits) / len(relevant), hits

precision, hits = precision_at_k(retrieved_chunks, relevant_chunks)
recall, _ = recall_at_k(retrieved_chunks, relevant_chunks)

print(f"Precision@5: {precision:.2f} ({len(hits)}/5)")
# Precision@5: 0.40 (2/5)
print(f"Recall@5: {recall:.2f} ({len(hits)}/3)")
# Recall@5: 0.67 (2/3)
print("Missed relevant chunk:", relevant_chunks - set(retrieved_chunks))
# Missed relevant chunk: {'chunk_9'}

chunk_9 is genuinely relevant, but it never made it into the top 5 — the model generating the answer has no way to use information it was never given, regardless of how well it reads the four chunks it did receive. Precision and recall often trade off against each other: try the three retriever qualities below and see how the missed chunks change.

Grounding: RAG Can Still Hallucinate

Perfect retrieval doesn’t guarantee a correct answer. A model can still misread a retrieved chunk, mix details from two different chunks into one incorrect claim, or answer using only the part of a chunk it “noticed” while ignoring a contradicting detail elsewhere in the same text. Grounding techniques push back against this directly: prompting the model to cite exactly which retrieved chunk supports each claim it makes, so a reader — or an automated check — can verify the answer against the actual source rather than trusting it by default.

Evaluating RAG: Faithfulness

RAGAS, a widely used open-source evaluation framework built specifically for RAG pipelines, formalizes this gap as faithfulness— evaluating a model’s ability to use retrieved passages faithfully, rather than just checking whether retrieval found the right chunks in the first place. It’s deliberately measured separately from retrieval quality: a system can score well on recall while still scoring poorly on faithfulness, if the model keeps misusing the context it’s actually given. This mirrors the same evaluation discipline covered in Comparing Ollama Models Advanced: measure the specific thing you actually care about, not a proxy that merely correlates with it.

Source: Es et al., “Ragas: Automated Evaluation of Retrieval Augmented Generation,” arXiv:2309.15217

Fun Fact

RAGAS was specifically designed to work without ground-truth human answers — it evaluates faithfulness, relevance, and retrieval quality using other models as judges, the same LLM-as-judge idea covered in Comparing Ollama Models Advanced, which is exactly what makes it practical to run continuously in production rather than only during a one-off manual evaluation.

Test Yourself