TinyStories
Ronen Eldan and Yuanzhi Li (Microsoft Research)
Millions of short stories in a three-year-old's vocabulary, used to show tiny models can write coherent text.
Short stories generated by GPT-3.5 and GPT-4 from a small vocabulary, described in a 2023 paper. One row is one story. Very small language models trained on it produce fluent, consistent narratives, making it a staple for studying small-model behavior.
Access
- Cost
- FreeNo charge.
- License
- Open licenseCDLA-Sharing 1.0
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet, Text
What's in it
- One row is
- One row per generated story
- Covers
- Not geographic
- Years
- 2023
- Updated
- One-time release
- Size
- 1 to 10 million stories
Quality
Good for research on small models and interpretability; narrow and repetitive for anything general.
- Stories are LLM-generated with limited variety
- Vocabulary and topics are deliberately narrow
- OpenAI output terms may apply to generated text
Connects to
- Cosmopedia: Larger-scale synthetic pretraining text that extends the idea
- FineWeb-Edu: Real-web educational corpus used for the opposite kind of small-model training
Questions it could answer
- How small can a model be and still tell a coherent story?
- What events and characters recur most often?