Skip to main content
The embedding model isn’t the only part of retrieval you can change. Two strategies are common in production RAG systems, and Truvec lets you test both on your own questions: hybrid search, which adds keyword search to vector search, and reranking, which re-scores the best candidates with a second model. Together with the embedding model, they form the retrieval pipeline of a run. A model with its pipeline is called a configuration, for example text-embedding-3-small - hybrid - rerank cohere/rerank-v3.5.

Dense search (the default)

Every run starts with dense search: the question is embedded and the vector database returns the chunks whose vectors are closest. This is what How RAG retrieval works describes, and it’s what a run does when you leave the pipeline settings untouched. Dense search is good at meaning (“I forgot my login” finds the password-reset article) and weaker at exact strings: product names, error codes, version numbers, part references. To an embedding model, ERR-4012 and ERR-4021 look almost the same. Hybrid search runs a keyword search next to the vector search and merges the two rankings.
  • The keyword search is BM25, the classic ranking used by search engines: chunks that contain the question’s rarer words, more often, rank higher. Compound terms such as ERR-404, v2.1.3 or gpt-4o are indexed whole and as their parts, so they match exactly.
  • The two rankings are merged with reciprocal rank fusion (RRF): each chunk gets points for its rank in each list, so a chunk that is high in both ends up first. Fusion uses ranks rather than scores, because cosine similarities and BM25 scores aren’t on the same scale.
  • The Keyword / Dense slider sets the weight of each ranking. 50% gives them equal say; move it toward Dense if your questions are mostly paraphrases, toward Keyword if they quote exact terms.
Hybrid search costs nothing extra: the keyword index is built by Truvec and doesn’t call any provider. It adds a few milliseconds of search time.
Hybrid search tends to help when users type identifiers, product or feature names, legal references or code symbols. It can hurt on conversational questions, where a frequent word pulls in the wrong chunks. That’s why it’s worth measuring rather than switching on by default.

Reranking

A reranker is a second model that reads the question and each candidate chunk together and scores how well the chunk answers it. Embedding models compare two vectors computed separately; a reranker (a cross-encoder) sees both texts at once, so it judges relevance more precisely, at the price of one call per question. How it works in Truvec:
  1. The first stage (dense or hybrid search) returns a pool of first-stage candidates: 30 by default, from 1 to 100, and never fewer than your top k.
  2. The reranker scores every candidate.
  3. The top k by reranker score become the result.
A reranker can only reorder what the first stage found. If the right chunk isn’t among the candidates, no reranker can bring it back: raise the number of candidates, or improve the first stage.

Available rerankers

Rerankers are included in every plan, like the embedding models: no provider API key needed.

What a reranker costs

A reranker usually costs more per query than embedding the question. With Cohere, every question is one search, so 1,000 queries cost $2.00 whatever the length of your chunks. With Voyage, the cost grows with the number of candidates and their length: 30 candidates of about 250 tokens is roughly 8,000 to 10,000 tokens per question, or about $0.40 to $0.50 per 1,000 queries with rerank-2.5. For comparison, embedding 1,000 short questions with text-embedding-3-small costs well under a cent. The cost per 1,000 queries of a run includes the reranker, so the trade-off is visible in the results. See Cost, speed and storage.
Cohere counts a candidate longer than about 500 tokens (question included) as several documents, and a search over more than 100 documents as several searches. Truvec’s estimate assumes one search per question. It’s accurate at the default 30 candidates and 1,000-character chunks, and an underestimate with close to 100 candidates and very long chunks.
Reranking also adds latency: one API round trip per question, typically a few hundred milliseconds. The results show it separately (rerank time and its p95) and in the total time per query.

Reading the results of a pipeline

In the Retrieval Inspector, the score next to each retrieved chunk depends on the pipeline: The movement is the most useful part: it tells you whether the right chunk was rescued by the keyword ranking, or pulled up from deep in the candidates by the reranker.

And chunking?

Chunking isn’t part of the pipeline, because it changes the chunks themselves. A golden set points at specific chunks, so a set built on one chunking can’t grade another. To compare chunkings, upload the documents with each setting and build a golden set on each. See Comparing chunking strategies.

When to try what

To find out what each change is worth, use a strategy comparison: one baseline, one change per configuration.