NaturalReasoning
Meta (Facebook AI Research)
Millions of hard questions back-translated from pretraining text and decontaminated.
Reasoning questions generated by working backward from documents in DCLM and FineMath, with reference answers extracted from those documents. One row is a question with its reference answer and, where available, a model response. Questions were decontaminated against MATH, GPQA, MMLU-Pro, and MMLU-STEM.
Access
- Cost
- FreeNo charge.
- License
- Open, non-commercial onlyCC BY-NC 4.0
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per question with a reference answer
- Covers
- Not geographic
- Years
- 2025
- Updated
- One-time release
- Size
- 1 to 10 million questions
Quality
Interesting construction with explicit decontamination, but reference answers are extracted automatically and may be wrong.
- Non-commercial license
- Answers extracted by a model from source text
- Questions can be underspecified
Connects to
- DCLM-baseline: One pretraining source the questions were generated from
- FineMath: Math pretraining source used for question generation
- GPQA: Benchmark used for decontamination
Questions it could answer
- Which fields are over-represented in back-translated questions?
- How often do reference answers match a stronger model's answer?