What’s Out There
← Back to swiping
Language & textFreeCompany

RedPajama-Data-V2

Together AI

Raw Common Crawl text in five languages with precomputed quality signals attached to each document.

Together released unfiltered web documents plus precomputed quality annotations, so researchers can set their own filters. One row is a document with those signals, and separate files hold duplicate-detection data. The README tables list tens of billions of documents and tens of trillions of estimated tokens.

Access

Cost
FreeNo charge; files are fetched with a custom loading script.
License
License not statedNot stated in the record; Common Crawl terms of use apply
Sign-up
No sign-up
How you get it
API and Bulk downloadJSON Lines, Parquet

What's in it

One row is
One row per web document, plus separate quality-signal and duplicate files
Covers
Global web in English, German, French, Spanish, and Italian
Years
Many Common Crawl snapshots (see README)
Updated
One-time release
Size
README totals: 32.9 billion documents and 50.6 trillion estimated tokens
raw contentquality signals (perplexity, line statistics)minhash duplicate IDslanguageurlcrawl snapshot

Quality

3 of 5

Useful for studying filtering because the signals are exposed, but the text itself is unfiltered and noisy.

  • Meant to be filtered before training
  • No explicit data license in the Hub record
  • Web text with unclear rights

Connects to

  • RedPajama-Data-1T: Earlier release that replicated the LLaMA training mix
  • FineWeb: Filtered web corpus; compares filtering choices against these raw signals

Questions it could answer

  • Which quality signals best predict a document being kept by common filters?
  • How much of each snapshot is duplicated across crawls?

Similar data sets

  • MADLAD-400

    A Common Crawl corpus in 419 languages whose lower-resource portions were manually audited.

    Language & textGoogle Research and Allen Institute for AIFree

  • FineWeb2

    Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

    Language & textHugging Face (FineWeb team)Free

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • FinePDFs

    About 3 trillion tokens extracted from PDFs, a source most web corpora skip because extraction is costly.

    Language & textHugging Face (FineWeb team)Free

  • DCLM-baseline

    A 4 trillion token pretraining set that won a controlled contest of data filtering recipes.

    Language & textDataComp for Language Models (mlfoundations)Free

  • Common Crawl

    A free, petabyte-scale copy of the public web, re-crawled again and again since 2008.

    Language & textCommon Crawl FoundationFree