FineWeb
Hugging Face
Trillions of tokens of cleaned English web text, with every filtering step published.
An English pretraining text set built from more than 100 Common Crawl snapshots (2013 to 2025) by extracting text, filtering for language and quality, and deduplicating. Unlike most commercial LLM training sets it is released openly along with the code and ablations behind each step, so it shows concretely what goes into a modern language model.
Access
- Cost
- FreeFree download from Hugging Face; sample subsets of 10B, 100B, and 350B tokens are available.
- License
- Open licenseODC-By 1.0; also subject to Common Crawl terms of use
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per web document (cleaned text with source URL and crawl ID)
- Covers
- English-language web
- Years
- 2013-2025 (Common Crawl snapshots CC-MAIN-2013-20 to CC-MAIN-2025-26)
- Updated
- Irregular
- Size
- About 25.9 billion documents; tens of terabytes; trillions of tokens
Quality
Well documented and openly benchmarked against other web sets; quality filters are heuristic and encode particular judgments about 'good' text.
- English only; FineWeb-2 covers other languages
- Filters can remove dialects and informal writing disproportionately
- Contains copyrighted text collected without author consent
Connects to
- Common Crawl: Raw crawl data that FineWeb filters and deduplicates
- The Pile: An earlier open LLM training mix to compare composition and filtering choices
Questions it could answer
- Which domains contribute the most tokens after filtering?
- How does the topic mix of the web text change between the 2013 and 2025 snapshots?