What is the difference between indexing-time and query-time in a RAG pipeline?

Indexing-time is the offline process of ingesting documents, chunking them, and embedding them into a vector store — it happens once per document (or whenever a document changes), and can be slow since no user is waiting on it. Query-time is what happens for every single user question: embedding the query, retrieving relevant chunks, and generating an answer — this needs to be fast, since a real user is waiting for the response.

← Back to the Introduction level

AI & ML — INTERMEDIATE

RAG Pipeline: Indexing vs Query-Time Flow, and Where Latency Comes From

The Introduction level walked through all four stages as one straight line. In practice, a RAG pipeline is really two separate processes running on very different clocks.

The Quick Answer

Ingest and embed together form the indexing-time flow — it runs once per document, offline, with no user waiting on it, so it can afford to take its time. Retrieve and generate form the query-time flow — it runs fresh for every single question, with a real user waiting for a response, so speed matters a great deal more here. And when that query-time flow does get measured, one stage usually dominates the total far more than the others.

Ingestion: Real Documents Are Messy

The ingest stage rarely receives clean, uniform text. A real document collection mixes PDFs, Markdown files, HTML pages, and more — each format needs its own extraction step before a document even reaches the chunking logic covered in Document Chunking Intermediate. A PDF’s multi-column layout, a Markdown file’s headers, and an HTML page’s nested tags all require different normalization before they’re reduced to the same plain, chunkable text.

Two Flows, Two Very Different Speeds

1
Indexing-time (ingest + embed). Runs once per document, or whenever a document changes. There’s no user actively waiting on this step, so a batch job processing thousands of documents overnight is a completely normal, acceptable design.
2
Query-time (retrieve + generate). Runs fresh for every question a real user asks. Every millisecond here is felt directly — this is the flow production systems actually optimize for speed.

Where the Latency Actually Goes

Breaking down a typical query-time flow into its stages reveals something not obvious from the pipeline diagram alone — the stages aren’t close to equal in cost:

Query embedding and vector search are both fast — often tens of milliseconds each, since a well-indexed vector search over hundreds of thousands of chunks doesn’t need to be slow. Generation is the real bottleneck: producing the final answer, token by token, typically takes far longer than every other stage in the pipeline combined. Optimizing RAG latency usually means optimizing generation speed first — the same territory covered in Quantization Advanced— since shaving milliseconds off retrieval barely moves the total.

Fun Fact

Because retrieval is such a small fraction of total latency, some production systems deliberately spend slightly more time on it — running a re-ranking pass to improve retrieval quality — since the extra time it costs is small compared to generation, but the quality improvement to the final answer can be significant.

Test Yourself

Why does it matter that indexing and query-time are treated as two separate processes?

Based on the illustrated latency breakdown, which stage typically accounts for the largest share of total RAG query latency?

What real-world document handling challenge does the ingestion stage have to deal with?

Ready for production concerns — keeping an index up to date, caching, a full worked pipeline in code, and a failure checklist across every stage? Continue to the Advanced level →