What’s Out There
← Back to swiping
Language & textFreeCompany

MADLAD-400

Google Research and Allen Institute for AI

A Common Crawl corpus in 419 languages whose lower-resource portions were manually audited.

Document-level text from every Common Crawl snapshot available as of August 2022. One row is one document labeled with a language. The creators audited samples by language, which they say makes it more reliable for rare languages than automatic-only corpora.

Access

Cost
FreeNo charge.
License
Open licenseODC-By 1.0
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet, JSON Lines

What's in it

One row is
One row per web document, organized by language
Covers
Global web, 419 languages
Years
Common Crawl through August 2022
Updated
One-time release
Size
Size category above 1 TB; token count not in the record
textlanguageurlclean/noisy variant

Quality

4 of 5

Rare example of manual language audits at scale, with a paper describing the method.

  • Snapshot ends in 2022
  • Web text with unclear rights
  • Audits cover samples, not every document

Connects to

  • CulturaX: Alternative multilingual corpus drawn from mC4 and OSCAR
  • FineWeb2: Newer multilingual corpus using fresher crawls

Questions it could answer

  • Which languages have the largest clean versus noisy gap?
  • How well do language labels hold up in a random audit?

Similar data sets

  • FineWeb2

    Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

    Language & textHugging Face (FineWeb team)Free

  • RedPajama-Data-V2

    Raw Common Crawl text in five languages with precomputed quality signals attached to each document.

    Language & textTogether AIFree

  • DCLM-baseline

    A 4 trillion token pretraining set that won a controlled contest of data filtering recipes.

    Language & textDataComp for Language Models (mlfoundations)Free

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • Common Crawl

    A free, petabyte-scale copy of the public web, re-crawled again and again since 2008.

    Language & textCommon Crawl FoundationFree

  • OpenAssistant Conversations (OASST1)

    161,443 chat messages written and ranked by volunteers in 35 languages, with no model in the loop.

    Language & textOpenAssistant (LAION-affiliated volunteers)Free