To compare chunking strategies, see Comparing chunking strategies: each chunking needs its own golden set, so it isn’t a setting of a comparison.
Compare models
1
Open Comparison and choose Compare models
2
Name it (optional)
For example “Small vs large on support docs”. Without a name, the date is used.
3
Select a golden dataset
Only committed datasets are listed.
4
Select at least two models
5
Set Top K and click Run Comparison
Each model is evaluated in turn in the background. The comparison shows how many are done and updates automatically.
Compare strategies
A strategy comparison starts from a baseline and adds configurations that each change one thing. Every configuration is then read against the baseline, so you know what each change is worth.1
Open Comparison and choose Compare strategies
2
Name it and select a golden dataset
For example “Does a reranker beat a bigger model?”.
3
Set up the baseline
The first card is the baseline: pick its embedding model, and leave its pipeline on Dense with No reranker to measure a plain setup. If you already run hybrid search or a reranker in production, set them here.
4
Add configurations
The buttons under the cards copy the baseline and change one thing:
- Hybrid search: the same model with BM25 keyword search fused in, at a 50/50 weight.
- Reranker: the same pipeline with a reranker on the first-stage candidates.
- Another model: the same pipeline with a different embedding model, for example to see whether a cheap model with a reranker beats a bigger model.
5
Set Top K and click Run Comparison
On the Free plan, a comparison holds up to 3 configurations. Each configuration counts toward that limit, whether it differs by its model or by its pipeline. A comparison counts as one evaluation run.
Reading a comparison
Select a comparison in Comparison History to open its results.The winner
Configurations are ranked by MRR, then by recall. The banner shows the top one and its scores. In a strategy comparison, it also says how it does against the baseline: the MRR difference, how many questions it finds that the baseline missed, and how many it misses that the baseline found. Treat a small lead with caution: on a small golden dataset, a difference of a few points can come down to one or two questions. See Choosing a configuration.Highlights
Cards show the cheapest to index, the cheapest per 1,000 queries, the fastest queries and the smallest index. When the best configuration is also the cheapest or fastest, the decision is easy. When it isn’t, weigh the difference.Results table
One row per configuration with hit rate, recall, precision, MRR and nDCG at your top k, latency, index cost, cost per 1,000 queries and vector storage. Latency covers the whole query: embedding the question, searching and reranking. Configurations that failed are marked Failed; hover the badge to see why. In a strategy comparison, two columns compare each configuration with the baseline:- the MRR difference, in green when it improves and red when it degrades,
- Fixed / broken: questions it finds that the baseline missed, and questions it misses that the baseline found. A change that fixes 4 questions and breaks 3 isn’t the same as one that fixes 1 and breaks none, even if the averages are close.
Quality vs cost
A model comparison shows Quality vs index cost: MRR against the cost to embed your corpus. A strategy comparison shows Quality vs query cost: MRR against the cost of 1,000 queries, because hybrid search and rerankers don’t change the index but do change what each query costs. Configurations in the top-left give the most quality for the money.Question by question
The Retrieval Inspector shows every configuration:- Configuration buttons switch between them. Each shows where that configuration ranked the right chunk for the current question (for example rank 1, or miss), so you can see at a glance where they disagree.
- With hybrid search, each chunk shows its rank in the dense and keyword rankings; with a reranker, its rank before reranking.
- The Hit rate by cut-off and Rank of the first relevant chunk charts overlay all configurations.
- The Cost & footprint and Separation & latency tables have one column per configuration.