Preparing documents for RAG: clean Markdown and heading-aware chunks

7 August 2026 · Milldown team · All posts

Most RAG guides spend their pages on chunk sizes and overlap percentages. Fair enough — but chunking strategy is the second step. If what you feed the chunker is raw PDF extraction full of page headers, broken reading order, and shattered tables, no overlap setting will save you. Retrieval quality is decided at pre-processing.

The pipeline

1. Convert everything to Markdown first.Markdown gives the chunker real structure to work with: headings mark section boundaries, tables stay atomic, lists stay together. Raw PDF text has none of that.
2. Clean before you chunk.Repeated page headers, footers, and page numbers land in the middle of paragraphs after extraction and end up embedded in your vectors. Strip them first — Milldown’s Clean Up pass does this fence-safely.
3. Chunk at heading boundaries, not character counts.A chunk that starts mid-sentence embeds poorly and reads worse when retrieved. Splitting on Markdown headings keeps each chunk a self-contained thought with its own title.
4. Carry metadata with every chunk.Source filename, section heading, and a content hash. When the model cites a chunk, you want to trace it to an exact source revision.
5. Export in the shape your loader expects.One JSONL record per chunk drops straight into most ingestion scripts; Markdown files with YAML front matter suit LlamaIndex-style directory loaders.

Why Markdown specifically

Two reasons. Structure: heading-aware splitters (LangChain’s MarkdownHeaderTextSplitter and friends) are the easiest reliable chunkers to operate, and they need Markdown headings to exist. Density: tokens spent on format noise are tokens not spent on content — in our measurements, Markdown versions of the same documents run 53–75% smaller than native formats, which means cheaper embeddings and more content per retrieved chunk.

Doing it without writing a parser

Milldown handles the front half of this pipeline on your Mac. Drop a folder of mixed PDFs, Word files, and decks (scans get OCR’d on-device), then use RAG chunk export: heading-aware chunks with YAML front matter carrying the source filename, section path, and a SHA-256 of the source — or JSONL, one chunk per record, ready for your embedding script. The user guide covers both shapes.

Sizing guidance that survives contact with reality

Start with one chunk per leaf section (whatever sits under a single heading), cap around 800–1,000 tokens, and only add overlap if your sections are long and unstructured. Then evaluate on your own queries — the best chunking config is corpus-specific, and a clean Markdown corpus makes re-chunking a five-minute experiment instead of a re-ingestion project.