Skip to main content

Do

Use real questions

Mix AI-generated questions with questions from support tickets, search logs or chat history. Real phrasing is what your system will face.

Review the golden dataset

Read every generated question and its expected chunks before committing. A wrong answer key produces wrong conclusions.

Match your production setup

Use the same chunking and the same top k as your RAG system. If you already have chunks, upload them as pre-chunked data.

Read the misses

Scores tell you how much, misses tell you why. Ten minutes in the Retrieval Inspector is worth more than another decimal of MRR.

Keep a baseline

Include your current setup in every comparison, as the baseline of a strategy comparison, so you always know whether a change is an improvement.

Change one thing at a time

In a strategy comparison, give each configuration one change from the baseline: a reranker, hybrid search or another model. To compare chunking, keep the configuration fixed and build a golden set on each chunking.

Avoid

With 20 questions, one question moves hit rate by 5 points. Treat close scores as ties, or add questions.
Questions that reuse the document’s exact words are easy for every model and hide real differences. Paraphrase, and include hard cases.
New chunks can answer existing questions without being listed as expected, which lowers scores unfairly. Upload all documents first, then build the dataset. Watch for the Corpus changed badge.
It biases retrieval toward one group of documents. Keep one set of settings per file type. Truvec warns you when you’re about to mix them.
Each model has its own scale, and hybrid search and rerankers produce different kinds of scores altogether. Compare rankings and metrics, not raw scores.
A reranker is billed on every query and adds a network round trip. Check the cost per 1,000 queries and the latency next to the quality gain. See What a reranker costs.
A golden set points at the chunks it was built on. After re-chunking, its expected chunks no longer exist and the scores are wrong. Build a new golden set on the new chunks.
A small quality gain rarely justifies a large increase in cost or latency. See Choosing a configuration.

A checklist before you decide

  • Documents are a representative sample, and all of them are uploaded.
  • Chunking matches production, with one set of settings per file type.
  • The golden dataset was reviewed, includes real questions and has 50+ questions for close calls.
  • Top k matches what your system sends to the LLM.
  • Your current setup is included as a baseline.
  • You read the misses of the top configurations, and the questions each change broke.
  • You weighed cost, latency and storage, not just quality.