trendingAI & ML — INTERMEDIATE
Chunking Strategies: Document Types, Recursive & Semantic Chunking
The Introduction level compared splitting by size against splitting at sentence boundaries. Real documents raise sharper questions: a Markdown file, a PDF, and a table each break in their own particular ways.
The Quick Answer
Different document types call for different chunking approaches — a Markdown file has headers worth splitting on, a table has rows that shouldn’t be split apart at all. Recursive chunking handles this gracefully by trying the most natural separator first (paragraphs) and only falling back to smaller ones (sentences, then words) when a piece is still too large. Semantic chunking goes further, using embedding similarity to find where a document’s topic actually shifts, rather than relying on any fixed rule at all. And whichever method is used, attaching metadata — source document, page number, section title — to every chunk is what makes a later citation possible.
Chunking Changes by Document Type
- 1
Markdown. Headers (
#,##,###) are a strong, explicit signal for where a document’s structure actually breaks — splitting on headers first, before falling back to plain paragraph splitting, tends to produce much more coherent chunks than ignoring that structure entirely. Every Markdown element needs its own treatment, covered in full below. - 2
PDFs. Multi-column layouts and embedded tables make naive top-to-bottom text extraction unreliable — text from two columns can get interleaved into a single, nonsensical chunk unless the extraction step is column-aware.
- 3
Tables. Splitting a table mid-row destroys its meaning entirely — a row usually only makes sense together, so table-aware chunking treats each row (or a small group of related rows) as an indivisible unit rather than applying a generic size-based rule.
Markdown Chunking Cheat Sheet: How to Chunk Every Markdown Element
A quick reference for how each Markdown element should be handled when chunking a Markdown document for RAG or an LLM — what to split on, and what to always keep intact.
| Markdown element | Chunking treatment | Why |
|---|---|---|
| # Heading 1 / ## Heading 2 / ### Heading 3 | Primary split point | Each header marks a genuine structural break — split here first, before falling back to paragraphs. |
| Paragraph text | Split on blank lines | Paragraphs are the next-largest natural unit once a header section is still too big to fit one chunk. |
| ```code block``` | Never split — keep whole | Splitting a code block mid-line breaks its syntax and makes it unreadable and unusable in a chunk. |
| - List item / 1. List item | Keep the list together | Individual list items usually only make sense in the context of the full list they belong to. |
| | Table | Row | | Never split mid-row | A table row split apart loses its meaning entirely — extract the whole table, header row included. |
| > Blockquote | Keep whole, treat like a paragraph | A blockquote is a single structural unit — parse it alongside headings and paragraphs, not mid-quote. |
| [Link](url) /  | Leave inline, don’t split around it | Links and images are part of the sentence or list item they sit in — splitting around them fragments that context. |
The pattern across every row is the same: split on the elements that mark a genuine structural break (headers, paragraph breaks), and never split through an element whose meaning or syntax depends on staying whole (code blocks, tables, list items, blockquotes).
Source: LangChain, “Text splitter integrations” documentation
Recursive Chunking
Recursive chunking tries a list of separators in order — largest first. It attempts to split on paragraph breaks; if a resulting piece is still larger than the target chunk size, it retries that specific piece using a smaller separator (sentence breaks), and continues shrinking the separator only for the pieces that actually need it. The result keeps naturally coherent units (a whole paragraph, a whole sentence) intact whenever possible, and only breaks them apart when there’s genuinely no other way to meet the size limit.
Semantic Chunking: Splitting by Meaning
Instead of any fixed rule, semantic chunking uses the same cosine similarity idea from the Embeddings series: embed each sentence, measure how similar each sentence is to the one right after it, and place a chunk boundary wherever that similarity drops sharply — a sharp drop is a strong signal that the topic just changed, regardless of how many words came before it.
Try it on a real example below — five sentences that shift from battery details to connectivity details partway through:
Metadata: Making Chunks Traceable
A chunk of text alone loses its origin the moment it’s stored. Attaching metadata — which document it came from, which page, which section heading — alongside the chunk means a retrieved answer can always be traced back to its exact source. This is precisely what makes the grounding and citation techniques from RAG Advanced possible in the first place — you can’t cite a source a chunk never recorded.
Source: LangChain, “Text splitter integrations” documentation
Fun Fact
Semantic chunkers still enforce hard minimum and maximum chunk sizes underneath the similarity logic — without a cap, a long, genuinely single-topic section could produce one enormous chunk that’s technically coherent but still too large to be useful.
Test Yourself
What does recursive chunking do differently from fixed-size chunking?
How does semantic chunking decide where to place a chunk boundary?
Why is it useful to attach metadata (source, page number, section title) to a chunk?
Ready for tables, code blocks, a real chunking pipeline in code, and the concrete failures that show up in production? Continue to the Advanced level →