What’s Out There
← Back to swiping
Language & textFreeCompany

FineWeb2

Hugging Face (FineWeb team)

Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

The multilingual successor to FineWeb: Common Crawl text cleaned and deduplicated for more than a thousand languages. One row is one web document with its text, language, script, and source URL. The authors validated the filtering with ablation experiments on nine diverse languages.

Access

Cost
FreeNo charge; one subset per language.
License
Open licenseODC-By 1.0
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per web document, grouped by language and script
Covers
Global web, over 1,000 languages
Years
Common Crawl snapshots (see card)
Updated
Irregular
Size
Not stated in the record; per-language sizes vary from tiny to billions of tokens
textlanguage codescripturlcrawl datelanguage score

Quality

4 of 5

Reproducible and carefully evaluated on a handful of languages. Quality for the long tail of languages is much less certain.

  • Language identification errors are likely for rare languages
  • Many languages have only a few thousand documents
  • Scraped web text with unclear rights

Connects to

  • FineWeb: English-only predecessor with the same filtering philosophy
  • CulturaX: Alternative multilingual web corpus covering 167 languages
  • MADLAD-400: Another Common Crawl multilingual corpus, with 419 languages

Questions it could answer

  • Which languages have enough data to train a model from scratch?
  • How does document length differ across scripts?

Similar data sets

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • FineWeb-Edu

    1.3 trillion tokens of web text kept because a classifier judged it educational.

    Language & textHugging Face (FineWeb team)Free

  • FinePDFs

    About 3 trillion tokens extracted from PDFs, a source most web corpora skip because extraction is costly.

    Language & textHugging Face (FineWeb team)Free

  • MADLAD-400

    A Common Crawl corpus in 419 languages whose lower-resource portions were manually audited.

    Language & textGoogle Research and Allen Institute for AIFree

  • FineMath

    34 billion tokens of math-teaching web pages, chosen by a classifier trained on Llama annotations.

    EducationHugging Face (SmolLM team)Free

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free