SQuAD (Stanford Question Answering Dataset)
Stanford University (Stanford NLP Group)
150,000 crowd-written questions on Wikipedia passages that defined a decade of reading benchmarks.
Reading-comprehension questions written by crowdworkers about paragraphs from 500+ Wikipedia articles, where each answer is a span of the passage. Version 2.0 added over 50,000 questions that cannot be answered from the text, to test whether systems know when to abstain. It is a landmark benchmark and a ready-made question-answer training set.
Access
- Cost
- FreeFree download; hidden test set is scored through the leaderboard.
- License
- Open licenseCC BY-SA 4.0
- Sign-up
- No sign-up
- How you get it
- Bulk downloadJSON
What's in it
- One row is
- One row per question, tied to a Wikipedia paragraph and its answer span
- Covers
- Not geographic
- Years
- Released 2016 (v1.1) and 2018 (v2.0)
- Updated
- One-time release
- Size
- About 150,000 questions in v2.0; training file about 40 MB
Quality
Carefully built and extremely widely used; now saturated, with models exceeding the human baseline.
- Benchmark is saturated; top scores beat the human baseline
- Questions often reuse the passage's wording, making them easier than real questions
- Text has almost certainly leaked into LLM training data
Connects to
- Wikipedia Database Dumps: Source articles for every passage
- GLUE Benchmark: Sibling benchmark suite covering other language-understanding tasks
Questions it could answer
- Which question types (who, when, why) remain hardest for models?
- How often do answerable questions copy words directly from the passage?