What is inter-annotator agreement?
Inter-annotator agreement is a measure of how often different reviewers give the same label to the same item.
Last reviewed
inter-annotator agreement in AI data
Low agreement signals unclear rubrics or unreliable reviewers.
Type: quality metric.
Where inter-annotator agreement applies on this site
- Data type: RLHF and preference data
- Guide: How to evaluate an AI data vendor
- Guide: How to buy AI training data
Quick answers
What is inter-annotator agreement?
Inter-annotator agreement is a measure of how often different reviewers give the same label to the same item.
Why does inter-annotator agreement matter in AI data?
Low agreement signals unclear rubrics or unreliable reviewers.