What’s Out There
← Back to swiping
Language & textFreeCompany

NaturalReasoning

Meta (Facebook AI Research)

Millions of hard questions back-translated from pretraining text and decontaminated.

Reasoning questions generated by working backward from documents in DCLM and FineMath, with reference answers extracted from those documents. One row is a question with its reference answer and, where available, a model response. Questions were decontaminated against MATH, GPQA, MMLU-Pro, and MMLU-STEM.

Access

Cost
FreeNo charge.
License
Open, non-commercial onlyCC BY-NC 4.0
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per question with a reference answer
Covers
Not geographic
Years
2025
Updated
One-time release
Size
1 to 10 million questions
questionreference answermodel responsessource corpus

Quality

3 of 5

Interesting construction with explicit decontamination, but reference answers are extracted automatically and may be wrong.

  • Non-commercial license
  • Answers extracted by a model from source text
  • Questions can be underspecified

Connects to

  • DCLM-baseline: One pretraining source the questions were generated from
  • FineMath: Math pretraining source used for question generation
  • GPQA: Benchmark used for decontamination

Questions it could answer

  • Which fields are over-represented in back-translated questions?
  • How often do reference answers match a stronger model's answer?

Similar data sets

  • OpenMathReasoning

    306,000 forum math problems with 3.2 million long chain-of-thought solutions.

    EducationNVIDIAFree

  • MMLU-Pro

    A harder MMLU with ten answer options and 12,000 reasoning-heavy questions.

    EducationTIGER-Lab (University of Waterloo and collaborators)Free

  • Llama-Nemotron Post-Training Dataset

    Millions of math, code, and reasoning examples used to post-train NVIDIA's Llama-Nemotron models.

    Language & textNVIDIAFree

  • OpenThoughts-114k

    114,000 long reasoning traces across math, science, code, and puzzles, released openly.

    Language & textOpen Thoughts teamFree

  • OpenR1-Math-220k

    220,000 math problems, each with two to four DeepSeek-R1 reasoning traces, many checked automatically.

    EducationHugging Face (Open-R1 project)Free

  • Humanity's Last Exam

    Thousands of expert-written questions designed to be too hard for current AI, across many fields.

    EducationCenter for AI Safety and Scale AIFree