Evaluation sets
An evaluation set is a collection of tasks with reference answers or rubrics, kept out of training, that measures how well a model performs.
Last reviewed
What evaluation sets contains
- Tasks and prompts
- Reference answers or graders
- Rubrics
- Difficulty and domain tags
Who buys it and how it is priced
| Attribute | Value |
|---|---|
| Belongs to | evaluation asset |
| Main buyers | Labs, enterprises and auditors that need trustworthy model measurement |
| Pricing | Priced per set or per project. Private, unreleased sets are worth more because public ones leak into training data. |
Quality criteria and risks
Quality criteria
- Held out from training
- Expert-written tasks
- Reliable grading
- Difficulty that separates models
Risks
- Contamination
- Ambiguous grading
- Saturation as models improve
Providers that list evaluation sets
Auction and pricing mechanisms that fit
- Posted-price sale: Fits catalogue datasets that can be sold many times, such as cloud data exchange listings.
- Reverse auction: Fits a buyer that knows what it needs and wants several vendors to compete, such as sourcing expert data from labs.
Key terms
- evaluation set
- An evaluation set is a collection of tasks used to measure a model, kept separate from training data.
- held-out set
- A held-out set is a portion of data deliberately excluded from training so it can test the model fairly.
- rubric
- A rubric is a written set of criteria that tells reviewers how to score or compare a model output.
Frequently asked questions
- Why do private evaluation sets cost more?
- A private set cannot be trained on by accident. Once it is public it loses much of its value as an honest test.