RedPajama-Data-V2
Together AI
Raw Common Crawl text in five languages with precomputed quality signals attached to each document.
Together released unfiltered web documents plus precomputed quality annotations, so researchers can set their own filters. One row is a document with those signals, and separate files hold duplicate-detection data. The README tables list tens of billions of documents and tens of trillions of estimated tokens.
Access
- Cost
- FreeNo charge; files are fetched with a custom loading script.
- License
- License not statedNot stated in the record; Common Crawl terms of use apply
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadJSON Lines, Parquet
What's in it
- One row is
- One row per web document, plus separate quality-signal and duplicate files
- Covers
- Global web in English, German, French, Spanish, and Italian
- Years
- Many Common Crawl snapshots (see README)
- Updated
- One-time release
- Size
- README totals: 32.9 billion documents and 50.6 trillion estimated tokens
Quality
Useful for studying filtering because the signals are exposed, but the text itself is unfiltered and noisy.
- Meant to be filtered before training
- No explicit data license in the Hub record
- Web text with unclear rights
Connects to
- RedPajama-Data-1T: Earlier release that replicated the LLaMA training mix
- FineWeb: Filtered web corpus; compares filtering choices against these raw signals
Questions it could answer
- Which quality signals best predict a document being kept by common filters?
- How much of each snapshot is duplicated across crawls?