What’s Out There
← Back to swiping
Language & textFreeCompany

SmolLM-Corpus

Hugging Face (SmolLM team)

The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

Three subsets chosen for training small language models: Cosmopedia v2 (synthetic textbooks and stories), FineWeb-Edu (deduplicated), and Python-Edu. Rows are documents, with each subset carrying its own fields.

Access

Cost
FreeNo charge.
License
Open licenseODC-By 1.0
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per document, in three subsets
Covers
Not geographic
Years
2024
Updated
One-time release
Size
100 million to 1 billion rows; Cosmopedia v2 has over 39 million synthetic documents
textprompt (Cosmopedia)audience (Cosmopedia)formatscore (FineWeb-Edu)

Quality

4 of 5

A tested recipe: a family of small models was trained on exactly this mix.

  • Cosmopedia text is LLM-generated and can repeat model errors
  • Generated text may carry the generating model provider's terms

Connects to

  • Cosmopedia: Synthetic subset, v2 of the original release
  • FineWeb-Edu: Source of the educational web subset

Questions it could answer

  • How does model quality change when synthetic text replaces web text?
  • Which Cosmopedia audience types are most common?

Similar data sets

  • Cosmopedia

    Synthetic textbooks, blog posts, and stories written by Mixtral to mimic what web pretraining text teaches.

    Language & textHugging Face (SmolLM team)Free

  • Dolma

    The three-trillion-token corpus behind OLMo, with the recipe and tooling published alongside it.

    Language & textAllen Institute for AIFree

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • MMMLU (Multilingual MMLU)

    The MMLU test set translated into 14 languages by professional human translators.

    EducationOpenAIFree

  • FineWeb2

    Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

    Language & textHugging Face (FineWeb team)Free

  • LMSYS-Chat-1M

    One million real conversations with 25 LLMs, collected from Vicuna demo and Chatbot Arena visitors.

    Language & textLMSYS (UC Berkeley and collaborators)Free