Skip to main content
Chunking is how your documents are split into the passages that retrieval returns. It is as important as the embedding model: the same model can score very differently depending on how the text was split. It also shapes what hybrid search and rerankers can do: keyword search matches words inside a chunk, and a reranker reads the whole chunk.

Why chunk size matters

Chunks too large

A chunk that covers several topics gets a “blurry” embedding that matches many questions a little and none of them well. It also sends more irrelevant text to the LLM.

Chunks too small

A chunk that holds a single sentence can lose its context: “It must be renewed every year” means little if you don’t know what “it” is.
A good chunk contains one complete idea with enough context to be understood on its own. For most prose, 500 to 1,500 characters (roughly 100 to 300 words) is a sensible range.

Overlap

Overlap repeats the end of one chunk at the start of the next. It prevents a sentence or an idea from being cut in half at a chunk boundary. An overlap of 10 to 20% of the chunk size is common, for example 200 characters for 1,000-character chunks.

Strategies available in Truvec

Not sure? Use Recursive with 1,000 characters and 200 overlap. It’s a robust default for most documents. Once you have a baseline, you can test other settings with their own golden set (see below).

A note on tokens

A token is the unit models read: roughly three quarters of an English word on average, so 1,000 tokens is about 750 words. Embedding models have a maximum input length measured in tokens, and providers bill by the token.

Consistency: one file type, one set of settings

Different file types can and often should use different strategies: Markdown by headers, source code by functions, prose with Recursive. That reflects how a real pipeline works. What you should avoid is splitting the same kind of document in two different ways within a project, for example half your PDFs at 1,000 characters and the other half at 200 tokens. Chunk size influences similarity scores, so retrieval becomes biased toward one group of documents and your results stop reflecting a real setup. Truvec helps you stay consistent:
  • The settings of your first upload become the project default, and the upload form starts from them.
  • If you upload a file type that already exists in the project with different settings, Truvec warns you before uploading. You can switch to the existing settings, keep yours on purpose, or cancel.
  • The Chunking settings in use table on the Data Upload page lists the settings used for each file type, and marks file types chunked in more than one way as Mixed.

Read your chunks

Before evaluating, look at what retrieval will actually see. On the Data Upload page, click any row of Ingested Chunks to read the full chunk with its source document. Check that chunks hold complete ideas, that tables and lists from PDFs came out readable, and that headers stayed with their sections.

Comparing chunking strategies

Testing whether a different chunking works better is a good idea. Unlike the embedding model, hybrid search or a reranker, chunking can’t be a setting of a comparison: a golden dataset points to specific chunks, so a dataset built on one chunking can’t grade another. Each chunking needs its own golden dataset.
1

Upload the same documents with the other chunking

In a new project, to keep both side by side (recommended), or in the same project after clearing the current documents.
2

Build a golden dataset on the new chunks

Generate it, or recreate the same questions by hand with the new chunk IDs. Keeping the same or similar questions makes the scores easier to compare.
3

Run the same evaluation in each project

Same configuration, same top k. Then compare the scores, and read the questions whose expected passage moved up or dropped out of the top k.
If you change the chunking in a project that already has committed golden datasets, Truvec warns you on the Data Upload page: those datasets point at the old chunks, and evaluating them on the new ones gives wrong scores. Generate a new golden set first.
Once you’ve picked a chunking, compare models, hybrid search and rerankers on it inside its project.

Already have chunks?

If your production pipeline already produces chunks, you can upload them as-is so Truvec evaluates exactly what you run in production. See Upload pre-chunked data.