Cosmopedia
Hugging Face (SmolLM team)
Synthetic textbooks, blog posts, and stories written by Mixtral to mimic what web pretraining text teaches.
An open synthetic pretraining dataset: Mixtral-8x7B-Instruct generated documents from prompts seeded with web samples and curated educational sources. One row is a prompt, the generated text, and the audience and format it targeted. Millions of rows in the v0.1 release.
Access
- Cost
- FreeNo charge.
- License
- Open licenseApache 2.0
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per generated document with its prompt
- Covers
- Not geographic
- Years
- 2024
- Updated
- One-time release
- Size
- 10 to 100 million rows
Quality
Documented and reproducible, but all text is model output and can contain factual errors and repetitive style.
- Hallucinated facts presented in textbook voice
- Topic coverage follows seed data, not the real distribution of knowledge
- Superseded by v0.2 inside SmolLM-Corpus
Connects to
- SmolLM-Corpus: Contains the improved Cosmopedia v2
- TinyStories: Earlier, much smaller synthetic corpus built on a similar idea
Questions it could answer
- Which prompt styles yield the most varied text?
- How repetitive are documents generated from the same seed topic?