HH-RLHF (Helpful and Harmless)
Anthropic
Human preference comparisons that helped define how chat assistants are trained to be helpful and harmless.
Two kinds of data from Anthropic's 2022 RLHF work: pairs of assistant replies where a human picked the better one on helpfulness or harmlessness, and red-team transcripts where crowdworkers tried to provoke harmful replies. One row is a chosen and a rejected conversation. The card says it is for preference models, not for supervised dialogue training.
Access
- Cost
- FreeNo charge.
- License
- Open licenseMIT
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadJSON Lines
What's in it
- One row is
- One row per pair of chosen and rejected conversations (preference data) or per red-team transcript
- Covers
- Not geographic
- Years
- 2022
- Updated
- One-time release
- Size
- 100,000 to 1 million rows across the preference and red-team subsets
Quality
Historically important and well described in a paper, but crowdworker judgments on an early model are noisy by modern standards.
- Contains offensive and harmful content by design
- Labeler agreement is modest and reflects a small crowd
- Responses came from early 2022 models
Connects to
- UltraFeedback: Newer, model-annotated preference set used for the same reward-model training
- HelpSteer2: Preference data with attribute ratings for reward modeling
- Chatbot Arena Conversations: Pairwise human votes on real chats from a public arena
Questions it could answer
- Which topics draw the most harmful completions in the red-team transcripts?
- How often do raters prefer the longer reply?