Chunking - Splitting Documents Correctly for RAG

Chunking means splitting long documents into smaller, self-contained text passages - the so-called chunks. It is the first and often underestimated step when building a RAG knowledge base. Only these passages are turned into embeddings and stored in a vector database, where they are searched later.

The reason is simple: a language model can only process a limited amount of text as context, and an embedding that presses an entire manual into a single vector blurs every nuance. Smaller passages can be embedded more precisely and retrieved more specifically.

Why Documents Are Split at All

• Targeted retrieval: The search should find exactly the paragraph that answers a question - not the whole document it sits in somewhere.
• Cleaner embeddings: A vector over a thematically self-contained passage carries a clear meaning. Averaged over a whole document, this sharpness is lost.
• Model context limit: Only a limited number of matching passages can be passed to the language model as evidence - compact chunks use this space efficiently.

Chunk Size and Overlap

The two most important levers are size and overlap:

• Chunk size: Passages that are too large mix several topics and dilute the embedding. Ones that are too small tear apart context, so a passage lacks the surroundings needed to be understandable. The right size depends on the type of documents - a contract is cut differently than a chat transcript.
• Overlap: So that a thought is not torn apart right at the cut, neighboring chunks share a small common text region. This keeps enough context at the edges, and a sentence running across a boundary is not lost.

Fixed vs. Semantic Chunking

1. Fixed chunking: The text is cut at a fixed length - every few hundred characters or after a certain number of tokens. Simple to implement, but blind to content: a cut can land mid-sentence or mid-argument.
2. Structure-based chunking: The cut follows the natural structure - headings, paragraphs, lists, tables. This keeps content units together.
3. Semantic chunking: Boundaries are drawn where the topic actually changes. The content similarity of consecutive passages is evaluated and a split is made where the meaning clearly jumps. More effort, but it yields thematically self-contained passages.

Impact on Retrieval Quality

Chunking helps decide whether a RAG answer is reliable. Poorly cut passages cause semantic search to find the wrong or an incomplete piece of evidence - the model then answers on a thin basis. Well cut chunks, by contrast, raise the hit probability and give the model a clean, complete context. That is why chunking is not a technical detail but a content decision that you align to the concrete documents and refine iteratively. Combined with hybrid search and reranking, hit quality can be secured further.

Chunking at Elasticbrains

At Elasticbrains we tune the splitting to your document types - from chunk size and overlap to whether fixed, structure-based, or semantic boundaries deliver the best retrieval quality. See how this becomes a verifiable knowledge base on our service page internal AI assistants / RAG knowledge base.