Expert demonstration data
Expert demonstration data is a record of a skilled professional completing a real task step by step, collected so a model can learn the process as well as the answer.
Last reviewed
What expert demonstration data contains
- Task brief and inputs
- Step-by-step expert workflow
- Final deliverable
- Rubric or scoring notes
Who buys it and how it is priced
| Attribute | Value |
|---|---|
| Belongs to | post-training data |
| Main buyers | Frontier labs building agents and professional-grade assistants |
| Pricing | Priced mainly by expert hourly rate, which tracks the profession: lawyers, physicians and quantitative analysts cost more than general contributors. |
Quality criteria and risks
Quality criteria
- Verified credentials
- Review by a second expert
- Realistic tasks
- Complete workflows rather than snippets
Risks
- Credential fraud
- Experts using models to generate answers
- Confidential client material leaking into samples
Providers that list expert demonstration data
Auction and pricing mechanisms that fit
- Posted-price sale: Fits catalogue datasets that can be sold many times, such as cloud data exchange listings.
- Reverse auction: Fits a buyer that knows what it needs and wants several vendors to compete, such as sourcing expert data from labs.
Key terms
- expert demonstration
- An expert demonstration is a recording of a skilled professional performing a task step by step, used to teach a model the process.
- rubric
- A rubric is a written set of criteria that tells reviewers how to score or compare a model output.
- reviewer calibration
- Reviewer calibration is the process of aligning reviewers' judgments with a gold standard through examples, feedback and checks.
- data provenance
- Data provenance is the documented history of where data came from, who created it and what rights attach to it.
Frequently asked questions
- Why do labs pay for expert demonstrations?
- Public web text shows finished work, not the steps behind it. Demonstrations capture the process that agents need to reproduce.