What’s Out There
← Back to swiping
Language & textFreeCompany

FineWeb

Hugging Face

Trillions of tokens of cleaned English web text, with every filtering step published.

An English pretraining text set built from more than 100 Common Crawl snapshots (2013 to 2025) by extracting text, filtering for language and quality, and deduplicating. Unlike most commercial LLM training sets it is released openly along with the code and ablations behind each step, so it shows concretely what goes into a modern language model.

Access

Cost
FreeFree download from Hugging Face; sample subsets of 10B, 100B, and 350B tokens are available.
License
Open licenseODC-By 1.0; also subject to Common Crawl terms of use
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per web document (cleaned text with source URL and crawl ID)
Covers
English-language web
Years
2013-2025 (Common Crawl snapshots CC-MAIN-2013-20 to CC-MAIN-2025-26)
Updated
Irregular
Size
About 25.9 billion documents; tens of terabytes; trillions of tokens
textURLCommon Crawl dumpdatelanguage scoretoken count

Quality

4 of 5

Well documented and openly benchmarked against other web sets; quality filters are heuristic and encode particular judgments about 'good' text.

  • English only; FineWeb-2 covers other languages
  • Filters can remove dialects and informal writing disproportionately
  • Contains copyrighted text collected without author consent

Connects to

  • Common Crawl: Raw crawl data that FineWeb filters and deduplicates
  • The Pile: An earlier open LLM training mix to compare composition and filtering choices

Questions it could answer

  • Which domains contribute the most tokens after filtering?
  • How does the topic mix of the web text change between the 2013 and 2025 snapshots?

Similar data sets

  • FineWeb2

    Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

    Language & textHugging Face (FineWeb team)Free

  • FineWeb-Edu

    1.3 trillion tokens of web text kept because a classifier judged it educational.

    Language & textHugging Face (FineWeb team)Free

  • FinePDFs

    About 3 trillion tokens extracted from PDFs, a source most web corpora skip because extraction is costly.

    Language & textHugging Face (FineWeb team)Free

  • FineMath

    34 billion tokens of math-teaching web pages, chosen by a classifier trained on Llama annotations.

    EducationHugging Face (SmolLM team)Free

  • DCLM-baseline

    A 4 trillion token pretraining set that won a controlled contest of data filtering recipes.

    Language & textDataComp for Language Models (mlfoundations)Free

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free