SmolLM-Corpus
Hugging Face (SmolLM team)
The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.
Three subsets chosen for training small language models: Cosmopedia v2 (synthetic textbooks and stories), FineWeb-Edu (deduplicated), and Python-Edu. Rows are documents, with each subset carrying its own fields.
Access
- Cost
- FreeNo charge.
- License
- Open licenseODC-By 1.0
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per document, in three subsets
- Covers
- Not geographic
- Years
- 2024
- Updated
- One-time release
- Size
- 100 million to 1 billion rows; Cosmopedia v2 has over 39 million synthetic documents
Quality
A tested recipe: a family of small models was trained on exactly this mix.
- Cosmopedia text is LLM-generated and can repeat model errors
- Generated text may carry the generating model provider's terms
Connects to
- Cosmopedia: Synthetic subset, v2 of the original release
- FineWeb-Edu: Source of the educational web subset
Questions it could answer
- How does model quality change when synthetic text replaces web text?
- Which Cosmopedia audience types are most common?