Document Chunking in Production: A Real Pipeline & Failure Cases

A Markdown-Aware Chunking Pipeline

Step one: split the document on headers, keeping each header attached to the section that follows it.

import re

def split_markdown_by_headers(text):
    pattern = r"(?=^#{1,3} )"
    sections = re.split(pattern, text, flags=re.MULTILINE)
    return [s.strip() for s in sections if s.strip()]

document = """# Installation

This guide covers installing the XR-400 sensor and its companion app.

## Setup Steps

Follow these steps in order to complete setup.

```bash
pip install xr400-sdk
xr400 init --device-id=001
xr400 calibrate
```

## Troubleshooting

If calibration fails, restart the device and try again.
"""

sections = split_markdown_by_headers(document)
print(f"Split into {len(sections)} header-based sections")
# Split into 3 header-based sections

Step two: for any section still too large on its own, fall back to splitting on blank lines — but only while tracking whether the parser is currently inside a code fence, so a chunk boundary is never inserted mid-block.

def protect_code_blocks(section, max_chars):
    if len(section) <= max_chars:
        return [section]

    lines = section.split("\n")
    chunks, current, current_len = [], [], 0
    in_code_block = False

    for line in lines:
        if line.strip().startswith("```"):
            in_code_block = not in_code_block
        current.append(line)
        current_len += len(line) + 1
        # Only allow a chunk break on a blank line, never mid-code-block.
        if not in_code_block and line.strip() == "" and current_len >= max_chars:
            chunks.append("\n".join(current).strip())
            current, current_len = [], 0

    if current:
        chunks.append("\n".join(current).strip())
    return chunks

# Forcing a small max_chars so it WOULD split mid-codeblock if unprotected:
chunks = protect_code_blocks(setup_section, max_chars=80)
for i, c in enumerate(chunks):
    fence_count = c.count("```")
    print(f"Chunk {i+1}: {fence_count} fences -> "
          f"{'OK (closed)' if fence_count % 2 == 0 else 'BROKEN'}")
# Chunk 1: 2 fences -> OK (closed)
# Chunk 2: 0 fences -> OK (closed)

Every chunk has an even number of code-fence markers — meaning every code block that starts also properly closes within the same chunk. That’s the entire point of tracking in_code_block: it turns off the ability to split at exactly the moments splitting would corrupt the document.

A Real Failure: Table Split Mid-Row

Here’s exactly what goes wrong when that same protection isn’t applied to a table, using naive fixed-size chunking on a real product comparison table:

table = """| Model | Battery | Range |
| --- | --- | --- |
| XR-400 | 18 months | 30m |
| XR-500 | 24 months | 45m |
| XR-600 | 12 months | 60m |
"""

def naive_fixed_chunks(text, size):
    return [text[i:i+size] for i in range(0, len(text), size)]

chunks = naive_fixed_chunks(table, 60)
for i, c in enumerate(chunks):
    print(f"--- Chunk {i+1} ---")
    print(repr(c))

# --- Chunk 1 ---
# '| Model | Battery | Range |\n| --- | --- | --- |\n| XR-400 | 1'
# --- Chunk 2 ---
# '8 months | 30m |\n| XR-500 | 24 months | 45m |\n| XR-600 | 12 '
# --- Chunk 3 ---
# 'months | 60m |\n'

Look closely at the XR-400’s battery figure: “18 months” gets torn apart into “1” at the end of chunk 1 and “8 months” at the start of chunk 2. A retrieval system that matches chunk 1 for a “XR-400 battery life” query would return a chunk ending in the meaningless fragment “XR-400 | 1” — the actual answer, “18 months,” is nowhere in it. The table’s header row is also now stuck at the top of chunk 1, disconnected from most of the data rows it labels.

The fix mirrors the code-block protection above: detect table boundaries (lines starting with |) and treat the whole table — header row included — as one indivisible unit, exactly like the Markdown chunking cheat sheet from the Intermediate level specifies.

Fun Fact

This exact class of bug — a number silently split across a chunk boundary — is especially dangerous precisely because it fails silently. The pipeline doesn’t crash or throw an error; it just quietly hands the RAG system a chunk that reads as plausible but is missing the one fact that mattered.

Test Yourself