Skip to main content
The configuration with the highest score isn’t automatically the right choice. This method weighs quality against cost and speed, and helps you avoid choosing on noise. It works the same whether you compare embedding models, or a model with and without hybrid search or a reranker.
1

Make sure the comparison is trustworthy

Before looking at the winner, check that:
  • the golden dataset has been reviewed and includes real user questions,
  • it has enough questions (50 or more for close decisions),
  • top k matches what your system sends to the LLM,
  • there is no Corpus changed badge on the comparison.
2

Pick your primary metric

Usually MRR, if your LLM relies mostly on the first chunks, or recall@k if answers often need several chunks. See Which metrics should I focus on?.
3

Decide what difference is meaningful

With 30 questions, one question is worth about 3 points of hit rate. A 2-point lead is likely noise. As a rule of thumb, treat configurations within a few points as tied unless your dataset is large. The question by question view tells you whether a lead comes from many questions or a lucky few. In a strategy comparison, Fixed / broken answers that directly: a change that fixes 5 questions and breaks 4 is a reshuffle, not an improvement.
4

Among the best, prefer the cheaper and faster

From the configurations that are best or tied for best on your primary metric, choose based on:
  • cost per 1,000 queries × your expected traffic, and index cost × how often you re-index,
  • latency per query, if users wait for answers in real time,
  • vector storage, if you’ll index millions of chunks.
The quality vs cost chart helps: the best trade-offs are in the top-left corner. Rerankers deserve a closer look here: they often add more to the cost and latency of a query than the embedding model does, so a reranker is worth it when its gain is clear, or when it lets you use a much cheaper embedding model.
5

Check the questions that matter most

Some questions are more important than others: legal, safety or billing questions, for example. In the Retrieval Inspector, check that your candidate configuration handles them.
6

Record the decision

The comparison stays in Reports as a record of which configurations were tested and which one won. Export it as a PDF report to share the decision and its evidence with your team.

Example

On 40 questions, a 0.02 MRR difference is within noise. Model B is 6.5 times cheaper per query and needs half the storage. Choose Model B, unless the question-by-question view shows Model A consistently wins on your most important questions. If Model A were at 0.90 MRR instead, the gap would likely be real and worth the cost for most use cases, because better retrieval directly means better answers.

With a reranker

Here the reranker gains 0.07 MRR, a real difference on 40 questions, and the Fixed / broken column shows whether it comes from many questions. It also multiplies the query cost by about 5,000 and adds 300 ms. At 10,000 queries a month that’s $20 a month for better answers, usually worth it. At 10 million, it’s $20,000, and a better embedding model alone may be the cheaper way to the same quality: add it as another configuration and compare.
Re-run the comparison when your documents change significantly, when new models or rerankers become available, or when you change how documents are chunked.