Skip to main content
A comparison runs two or more configurations on the same golden dataset, with the same top k, so their results are directly comparable. A configuration is an embedding model with its retrieval pipeline: dense or hybrid search, with or without a reranker (see Hybrid search and reranking). The Comparison page has two modes: To compare chunking strategies, see Comparing chunking strategies: each chunking needs its own golden set, so it isn’t a setting of a comparison.

Compare models

1

Open Comparison and choose Compare models

2

Name it (optional)

For example “Small vs large on support docs”. Without a name, the date is used.
3

Select a golden dataset

Only committed datasets are listed.
4

Select at least two models

5

Set Top K and click Run Comparison

Each model is evaluated in turn in the background. The comparison shows how many are done and updates automatically.
Include your current model, if you have one, as a baseline. The question is rarely “which model is best?” but “is another model better enough to justify switching?”

Compare strategies

A strategy comparison starts from a baseline and adds configurations that each change one thing. Every configuration is then read against the baseline, so you know what each change is worth.
1

Open Comparison and choose Compare strategies

2

Name it and select a golden dataset

For example “Does a reranker beat a bigger model?”.
3

Set up the baseline

The first card is the baseline: pick its embedding model, and leave its pipeline on Dense with No reranker to measure a plain setup. If you already run hybrid search or a reranker in production, set them here.
4

Add configurations

The buttons under the cards copy the baseline and change one thing:
  • Hybrid search: the same model with BM25 keyword search fused in, at a 50/50 weight.
  • Reranker: the same pipeline with a reranker on the first-stage candidates.
  • Another model: the same pipeline with a different embedding model, for example to see whether a cheap model with a reranker beats a bigger model.
Each card can then be edited: model, Retrieval (Dense or Hybrid, with the Keyword / Dense weight), Reranker, and First-stage candidates (30 by default). You can combine changes in one card, for example hybrid search and a reranker, but a card that changes one thing is easier to read.
5

Set Top K and click Run Comparison

On the Free plan, a comparison holds up to 3 configurations. Each configuration counts toward that limit, whether it differs by its model or by its pipeline. A comparison counts as one evaluation run.
Hybrid search costs nothing extra. Rerankers are billed by their provider per search or per token, and that cost is added to the cost per 1,000 queries of the configuration. See What a reranker costs.

Reading a comparison

Select a comparison in Comparison History to open its results.

The winner

Configurations are ranked by MRR, then by recall. The banner shows the top one and its scores. In a strategy comparison, it also says how it does against the baseline: the MRR difference, how many questions it finds that the baseline missed, and how many it misses that the baseline found. Treat a small lead with caution: on a small golden dataset, a difference of a few points can come down to one or two questions. See Choosing a configuration.

Highlights

Cards show the cheapest to index, the cheapest per 1,000 queries, the fastest queries and the smallest index. When the best configuration is also the cheapest or fastest, the decision is easy. When it isn’t, weigh the difference.

Results table

One row per configuration with hit rate, recall, precision, MRR and nDCG at your top k, latency, index cost, cost per 1,000 queries and vector storage. Latency covers the whole query: embedding the question, searching and reranking. Configurations that failed are marked Failed; hover the badge to see why. In a strategy comparison, two columns compare each configuration with the baseline:
  • the MRR difference, in green when it improves and red when it degrades,
  • Fixed / broken: questions it finds that the baseline missed, and questions it misses that the baseline found. A change that fixes 4 questions and breaks 3 isn’t the same as one that fixes 1 and breaks none, even if the averages are close.

Quality vs cost

A model comparison shows Quality vs index cost: MRR against the cost to embed your corpus. A strategy comparison shows Quality vs query cost: MRR against the cost of 1,000 queries, because hybrid search and rerankers don’t change the index but do change what each query costs. Configurations in the top-left give the most quality for the money.

Question by question

The Retrieval Inspector shows every configuration:
  • Configuration buttons switch between them. Each shows where that configuration ranked the right chunk for the current question (for example rank 1, or miss), so you can see at a glance where they disagree.
  • With hybrid search, each chunk shows its rank in the dense and keyword rankings; with a reranker, its rank before reranking.
  • The Hit rate by cut-off and Rank of the first relevant chunk charts overlay all configurations.
  • The Cost & footprint and Separation & latency tables have one column per configuration.
The most useful questions to read are the ones where configurations disagree: one finds the right chunk at rank 1 and another misses it. In a strategy comparison, start with the questions a change broke.

Comparisons and evaluations

Each configuration in a comparison is a regular evaluation behind the scenes, but comparison runs aren’t listed in Evaluation History, to keep it readable. Deleting a comparison deletes its evaluations. Embeddings are shared: configurations that use the same model embed your chunks once, so adding hybrid search or a reranker to a model you’ve already evaluated only embeds the questions.