RLHF and preference data
RLHF and preference data is a set of human judgments that rank or compare model outputs, used to train reward models and align language models with what people prefer.
Last reviewed
What rlhf and preference data contains
- Prompt with two or more model responses
- Human ranking or pairwise choice
- Written rationale or rubric scores
- Annotator metadata
Who buys it and how it is priced
| Attribute | Value |
|---|---|
| Belongs to | post-training data |
| Main buyers | Frontier labs and any team that post-trains a model |
| Pricing | Priced per comparison or per hour of reviewer time. Expert or long-form comparisons cost far more than short crowd judgments. |
Quality criteria and risks
Quality criteria
- Inter-annotator agreement
- Reviewer calibration against gold examples
- Clear rubrics
- Diversity of prompts
Risks
- Reviewer bias baked into the reward model
- Low-effort or model-generated judgments
- Unclear ownership of annotator output
Providers that list rlhf and preference data
Auction and pricing mechanisms that fit
- Posted-price sale: Fits catalogue datasets that can be sold many times, such as cloud data exchange listings.
- Reverse auction: Fits a buyer that knows what it needs and wants several vendors to compete, such as sourcing expert data from labs.
Key terms
- RLHF
- RLHF, or reinforcement learning from human feedback, is a training method in which human preference judgments are used to train a reward model that then guides a language model.
- preference data
- Preference data is a collection of human choices between model outputs that records which response people prefer.
- inter-annotator agreement
- Inter-annotator agreement is a measure of how often different reviewers give the same label to the same item.
- rubric
- A rubric is a written set of criteria that tells reviewers how to score or compare a model output.
Frequently asked questions
- What is the difference between RLHF data and SFT data?
- RLHF data ranks outputs, so a model learns which answer people prefer. SFT data shows one good answer, so a model learns to imitate it.
- How is RLHF data priced?
- Most vendors charge per task or per reviewer hour, with expert-level comparisons priced well above general-crowd work. Published rate cards are rare.