What is document chunking and why is it necessary?

Document chunking is the process of splitting a large document into smaller, manageable pieces before it can be searched or given to a language model. It’s necessary because a model’s context window has a fixed limit, and retrieval systems work by matching a query against individual pieces of a document rather than the entire thing at once — an unsplit, whole document is both too large to fit and too coarse to search effectively.

AI & ML — INTRODUCTION

Document Chunking: How Big Documents Get Split for an LLM

A 100-page manual, a legal contract, a codebase’s documentation — none of it can go to a model as one single block of text. Document chunking is how it gets broken into pieces a model can actually work with.

The Quick Answer

Chunking splits a large document into smaller, self-contained pieces before it’s searched or handed to a model — necessary because a context window has a hard size limit, and RAG retrieval needs individual, targeted pieces to match a query against rather than one giant, undifferentiated document. How big the pieces are matters, but so does where the split happens — cutting through the middle of a sentence produces a fundamentally worse chunk than cutting cleanly between two of them, even at the exact same chunk size.

What the Experts Say

LangChain’s own reference documentation for its widely used text-splitting tool describes its core approach plainly: recursively trying different characters — larger separators like paragraph breaks first, falling back to smaller ones like sentences or words — until it finds one that successfully produces chunks within the target size. That recursive, try-larger-separators-first approach is precisely why boundary-aware splitting tends to outperform naive fixed-size splitting in practice. See LangChain, “RecursiveCharacterTextSplitter” reference docs.

Three Basic Chunking Approaches

1
Fixed-size chunking. Split the text every N characters or words, regardless of what’s at that exact cut point. Simple and predictable, but it can cut a sentence — or even a word — clean in half.
2
Sentence-boundary chunking. Only split where a sentence actually ends. Every chunk is a complete thought, even if chunk sizes end up varying more than with a fixed-size approach.
3
Paragraph/section-boundary chunking. Split at paragraph or section breaks — larger, naturally coherent units, useful when a single sentence alone wouldn’t carry enough context to be useful on its own.

See the Difference Yourself

Same sample text, split two different ways at roughly the same chunk size — watch what happens right at the boundary.

Fun Fact

Boundary-aware chunking doesn’t require anything fancy under the hood — a reasonably good sentence splitter just needs to avoid firing on things that look like sentence ends but aren’t, like the decimal point in “Bluetooth 5.2,” which is exactly the kind of edge case that trips up naive splitting logic.

Test Yourself

Why can’t a large document just be sent to a model as one single piece?

What is the main risk of fixed-size chunking?

Ready for how chunking changes across real document types — Markdown, PDFs, and tables — and what recursive and semantic chunking actually look like? Continue to the Intermediate level →