What’s Out There
← Back to swiping
Language & textFreeAcademic

DCLM-baseline

DataComp for Language Models (mlfoundations)

A 4 trillion token pretraining set that won a controlled contest of data filtering recipes.

The best-performing baseline from the DataComp-LM benchmark, which holds model training fixed and varies the data. It is Common Crawl text filtered by a learned quality classifier. One row is one document with text and metadata.

Access

Cost
FreeNo charge.
License
Open licenseCC BY 4.0
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet, JSON Lines

What's in it

One row is
One row per web document
Covers
Global web, English
Years
Common Crawl snapshots
Updated
One-time release
Size
About 4 trillion tokens in roughly 3 billion documents
texturllanguagefasttext quality scorecrawl info

Quality

4 of 5

Backed by a controlled comparison across many filtering strategies, which most corpora lack.

  • Optimized for benchmark scores, which may favor trivia and exam-like text
  • Web text with unclear rights

Connects to

  • FineWeb-Edu: Rival filtered web corpus compared in open-model ablations
  • Natural Reasoning: Questions in that set were back-translated from DCLM text

Questions it could answer

  • What does a classifier-selected web corpus look like compared with the raw crawl?
  • Which domains dominate the retained documents?

Similar data sets

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • MADLAD-400

    A Common Crawl corpus in 419 languages whose lower-resource portions were manually audited.

    Language & textGoogle Research and Allen Institute for AIFree

  • FineWeb2

    Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

    Language & textHugging Face (FineWeb team)Free

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free

  • Common Crawl

    A free, petabyte-scale copy of the public web, re-crawled again and again since 2008.

    Language & textCommon Crawl FoundationFree

  • FineWeb-Edu

    1.3 trillion tokens of web text kept because a classifier judged it educational.

    Language & textHugging Face (FineWeb team)Free