What is RAG (Retrieval-Augmented Generation) and how does it reduce hallucination?

RAG is a technique where relevant documents are retrieved and added to a model’s context before it answers, so the model can base its response on real, provided text instead of only what it memorized during training. This reduces hallucination because the model has less need to guess or invent facts when the actual source material is right in front of it — though it doesn’t eliminate hallucination entirely, since a model can still misread or misuse the retrieved text.

AI & ML — INTRODUCTION

What Is RAG? How Retrieval-Augmented Generation Reduces Hallucination

Ask a model about your company’s internal policy, or a product it’s never seen documentation for, and it will often answer anyway — confidently, and sometimes wrongly. RAG is the fix: give the model the actual document to read first.

The Quick Answer

Retrieval-Augmented Generation (RAG) retrieves relevant documents and adds them to a model’s context before it answers, instead of letting the model rely only on whatever it memorized during training. Think of it as the difference between a closed-book exam and an open-book one — a model without RAG has to answer from memory alone, while a model with RAG gets to actually read the relevant material first. That’s why RAG reduces hallucination: the model has real text in front of it, so it has less need to guess or invent an answer that merely sounds plausible.

What the Experts Say

The technique gets its name from the paper that introduced it, which combined a model’s own trained knowledge with retrieval from an external, non-parametric source. Its own evaluation found that these retrieval-augmented models generate more specific, diverse, and factual language than an equivalent model without retrieval — a direct research result behind why RAG became the standard fix for hallucination on knowledge-intensive questions. See Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” arXiv:2005.11401.

How It Actually Works

1
A question comes in. The system receives a query — the same as any normal question to an AI assistant.
2
Relevant documents get retrieved. Using embeddings and a vector database, the system searches a document collection for the pieces of text most related to the question.
3
Retrieved text gets added to the context. Those documents are inserted into the model’s context window, alongside the original question — the model now has real source material to work with.
4
The model answers using that context. Instead of relying purely on patterns memorized during training, the model generates its answer grounded in the specific text it was just given.

Steps 2 and 3 lean directly on two things already covered in this series: embeddings turn text into a form that can be searched by meaning, and a vector database is what actually performs that search at scale. RAG is what those two pieces are built to support.

See the Difference Yourself

Pick a question a model was never specifically trained on, and compare its answer with and without a retrieved document to ground it.

Fun Fact

RAG doesn’t require retraining the model at all — the same underlying model can answer questions about a document collection it has never been trained on, simply by having the relevant pieces handed to it at the moment it answers.

Test Yourself

What does RAG add to a model’s process before it answers a question?

Does RAG completely eliminate hallucination?

Ready for the engineering behind retrieval — how documents actually get split up, and what happens when the wrong piece gets retrieved? Continue to the Intermediate level →