DCLM-baseline
DataComp for Language Models (mlfoundations)
A 4 trillion token pretraining set that won a controlled contest of data filtering recipes.
The best-performing baseline from the DataComp-LM benchmark, which holds model training fixed and varies the data. It is Common Crawl text filtered by a learned quality classifier. One row is one document with text and metadata.
Access
- Cost
- FreeNo charge.
- License
- Open licenseCC BY 4.0
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet, JSON Lines
What's in it
- One row is
- One row per web document
- Covers
- Global web, English
- Years
- Common Crawl snapshots
- Updated
- One-time release
- Size
- About 4 trillion tokens in roughly 3 billion documents
Quality
Backed by a controlled comparison across many filtering strategies, which most corpora lack.
- Optimized for benchmark scores, which may favor trivia and exam-like text
- Web text with unclear rights
Connects to
- FineWeb-Edu: Rival filtered web corpus compared in open-model ablations
- Natural Reasoning: Questions in that set were back-translated from DCLM text
Questions it could answer
- What does a classifier-selected web corpus look like compared with the raw crawl?
- Which domains dominate the retained documents?