What’s Out There
← Back to swiping
Language & textFreeCompany

TinyStories

Ronen Eldan and Yuanzhi Li (Microsoft Research)

Millions of short stories in a three-year-old's vocabulary, used to show tiny models can write coherent text.

Short stories generated by GPT-3.5 and GPT-4 from a small vocabulary, described in a 2023 paper. One row is one story. Very small language models trained on it produce fluent, consistent narratives, making it a staple for studying small-model behavior.

Access

Cost
FreeNo charge.
License
Open licenseCDLA-Sharing 1.0
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet, Text

What's in it

One row is
One row per generated story
Covers
Not geographic
Years
2023
Updated
One-time release
Size
1 to 10 million stories
story textvocabulary-limited prosegenerated by GPT-3.5 or GPT-4train splitvalidation split

Quality

3 of 5

Good for research on small models and interpretability; narrow and repetitive for anything general.

  • Stories are LLM-generated with limited variety
  • Vocabulary and topics are deliberately narrow
  • OpenAI output terms may apply to generated text

Connects to

  • Cosmopedia: Larger-scale synthetic pretraining text that extends the idea
  • FineWeb-Edu: Real-web educational corpus used for the opposite kind of small-model training

Questions it could answer

  • How small can a model be and still tell a coherent story?
  • What events and characters recur most often?

Similar data sets

  • UltraChat 200k

    A 200,000-dialogue filtered slice of ChatGPT-generated conversations that trained Zephyr-7B.

    Language & textHugging Face H4 team (filtered from Tsinghua's UltraChat)Free

  • NuminaMath-CoT

    About 860,000 math problems with chain-of-thought solutions, from Chinese high school to olympiad level.

    EducationProject NuminaFree

  • Llama-Nemotron Post-Training Dataset

    Millions of math, code, and reasoning examples used to post-train NVIDIA's Llama-Nemotron models.

    Language & textNVIDIAFree

  • FineVision

    17.3 million images and 24.3 million samples merged from many sources to train open vision-language models.

    Images & videoHugging Face (M4 team)Free

  • FineWeb2

    Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

    Language & textHugging Face (FineWeb team)Free

  • YouTube-Commons

    Transcripts of over 2 million YouTube videos, all posted under a CC-BY license, about 45 billion words.

    Social mediaPleiasFree