You need documents in the project before creating a dataset. See Upload documents.
1. Generate the first questions
1
Open Golden Set and click Create Dataset
2
Describe the dataset
Enter a name and an optional description, for example “Customer questions, v1”.
3
Choose the number of questions
Up to 20 on Free, 100 on Starter and 200 on Team Pro. Start with 20 to 50. You’ll review every question, so generate a number you can realistically check.
4
Click Generate Dataset
Truvec sends your chunks to an AI model in batches, spread across the whole corpus, and asks for realistic questions answered by those chunks, with one to three expected chunks each. Malformed answers from the AI are retried automatically. Generation takes from a few seconds to a minute or two.
2. Review every question
Open the dataset. Each row shows a question, its expected chunks and a status.- Click a chunk row to read the full chunk. Use the arrows to step through all the expected chunks of the question. Hovering a chip shows a preview.
- Check the status column. Valid means all expected chunks exist. Missing chunk means one of them was deleted with its document.
Do the expected chunks really answer it?
Do the expected chunks really answer it?
If not, edit the chunk list or delete the question.
Does another chunk also answer it?
Does another chunk also answer it?
Add it to the expected chunks. Otherwise, a model that retrieves it is counted as wrong. Use the Playground to search for other chunks that answer the question.
Would a real user phrase it this way?
Would a real user phrase it this way?
Rewrite questions that copy the document’s wording or sound unnatural.
Is it too vague?
Is it too vague?
“What is the policy?” could match dozens of chunks. Make it specific or delete it.
3. Add your own questions
Click Add Entry, type the question and enter the expected chunk IDs separated by commas, for example14, 15. To find chunk IDs, search the Ingested Chunks table on the Data Upload page (the ID column), or run the question in the Playground, which shows the ID of every chunk it retrieves.
4. Commit the dataset
When the questions are right, click Commit Dataset. The dataset becomes read-only and appears in the Evaluation and Comparison pages. Only committed datasets can be evaluated. This ensures that results from different runs are graded on exactly the same questions.Export a golden dataset
To keep a copy of a dataset or reuse it in your own tests, click Export JSON on the dataset’s page, or the download icon on its row in the list of golden sets. Drafts and committed datasets can both be exported. The file contains the dataset’s name, description and status, and every question with its expected chunks:customer-support-queries.json