FineWeb2
Hugging Face (FineWeb team)
Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.
The multilingual successor to FineWeb: Common Crawl text cleaned and deduplicated for more than a thousand languages. One row is one web document with its text, language, script, and source URL. The authors validated the filtering with ablation experiments on nine diverse languages.
Access
- Cost
- FreeNo charge; one subset per language.
- License
- Open licenseODC-By 1.0
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per web document, grouped by language and script
- Covers
- Global web, over 1,000 languages
- Years
- Common Crawl snapshots (see card)
- Updated
- Irregular
- Size
- Not stated in the record; per-language sizes vary from tiny to billions of tokens
Quality
Reproducible and carefully evaluated on a handful of languages. Quality for the long tail of languages is much less certain.
- Language identification errors are likely for rare languages
- Many languages have only a few thousand documents
- Scraped web text with unclear rights
Connects to
- FineWeb: English-only predecessor with the same filtering philosophy
- CulturaX: Alternative multilingual web corpus covering 167 languages
- MADLAD-400: Another Common Crawl multilingual corpus, with 419 languages
Questions it could answer
- Which languages have enough data to train a model from scratch?
- How does document length differ across scripts?