Chunking and retrieval

Chunking is how you cut your documents into pieces before embedding them. It sounds trivial and it is where most RAG systems are won or lost. The quality of your chunks decides the quality of every answer.

Why chunking matters

An embedding represents one chunk of text. If the chunk is too big, its meaning is diluted and it matches too many things vaguely. If it is too small, it lacks the context to mean anything at all. The right chunk is the smallest piece that still carries complete meaning.

The knobs

1. Size — the number of tokens per chunk. A common range is a few hundred to a couple thousand tokens.
2. Overlap — how much the chunks share at the edges, so a sentence is not split mid-thought. A little overlap (10–20%) prevents losing context at the boundary.
3. Structure — whether you respect the document's own boundaries (headings, paragraphs, tables) or cut by a fixed window. Respecting structure almost always wins.

The failures to avoid

• Splitting a definition from its term. The chunk has the term, the next chunk has the definition. Neither is useful alone.
• One giant chunk. Everything matches it, so retrieval returns it constantly and the model drowns in irrelevant text.
• Chunks with no metadata. You need to know where a chunk came from, both to cite and to filter.

The practical method

1. Start with the document's structure: split by headings and paragraphs, not by a blind character count.
2. Keep chunks on the smaller side, with a little overlap.
3. Add metadata: source, section, page.
4. Test retrieval in isolation before you build the whole pipeline: ask questions and look at what got retrieved, not just the final answer.

Chunking is where RAG stops being magic and becomes engineering. Do it deliberately, and the rest of the system is much easier.