trendingAI & ML — INTERMEDIATE
RAG Engineering: Chunking Strategy & Re-ranking
The Introduction level treated retrieval as a black box: a question goes in, the right document comes out. In practice, how that document gets split up before it’s even stored decides whether the right piece can be found at all.
The Quick Answer
Before documents can be embedded and searched, they get split into smaller pieces called chunks — an entire document embedded as one unit loses the ability to retrieve just the specific relevant part. Chunk size is a real trade-off: too small, and a single fact can get split across a boundary; too large, and irrelevant text dilutes the chunk’s relevance to any one query. Many production pipelines also add a re-ranking step after initial retrieval — a slower but more accurate second pass that re-scores a larger candidate set before handing the final few chunks to the model.
Chunk Size: See the Trade-off
Here’s a real product spec paragraph, split at different chunk sizes. Watch what happens to the battery-life fact as the chunk size shrinks.
Common practical starting points: around 256–512 tokens for chunks meant to answer specific, factoid-style questions, and 1,000+ tokens (or whole sections) for chunks meant to support broader, more analytical questions that need surrounding context to make sense.
Overlap: Not as Obviously Useful as It Sounds
A common recommendation is to make consecutive chunks overlap slightly — say, the last 50–100 tokens of one chunk repeated at the start of the next — specifically to avoid the boundary problem shown above. It’s a reasonable idea, but it isn’t free: it means storing and searching through more total text than the source document actually contains. A 2026 systematic study testing this directly found that overlap provided no measurable benefit to retrieval quality in their evaluation, while still increasing indexing cost — a useful reminder to test an assumption like this against your own data rather than applying it automatically.
Re-ranking: A Second, More Careful Pass
Initial retrieval — the vector database search from the Introduction level — is fast but approximate: it typically pulls back a fairly large candidate set (say, 50–100 chunks) rather than trying to get the final ranking perfect in one pass. Re-ranking adds a second, slower step: a model that looks at the actual query and each candidate chunk together, scoring their relevance more precisely than the initial search could, and reorders the candidates before only the top few are handed to the generating model. It costs extra time per query, but consistently improves the quality of what actually reaches the model — which matters more than raw retrieval speed for most RAG applications.
Fun Fact
The same 2026 chunking study found something called a “context cliff” — retrieval quality dropping noticeably once the total retrieved context grows past roughly 2,500 tokens, echoing the same idea covered in Context Engineering Advanced’s “context rot”: more retrieved text isn’t automatically better.
Test Yourself
What can go wrong if chunks are too small?
What did a 2026 systematic study find about chunk overlap?
What does a re-ranking step do in a RAG pipeline?
Ready to see what happens when the wrong chunk gets retrieved anyway, and how to measure that with real precision/recall code? Continue to the Advanced level →