MADLAD-400
Google Research and Allen Institute for AI
A Common Crawl corpus in 419 languages whose lower-resource portions were manually audited.
Document-level text from every Common Crawl snapshot available as of August 2022. One row is one document labeled with a language. The creators audited samples by language, which they say makes it more reliable for rare languages than automatic-only corpora.
Access
- Cost
- FreeNo charge.
- License
- Open licenseODC-By 1.0
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet, JSON Lines
What's in it
- One row is
- One row per web document, organized by language
- Covers
- Global web, 419 languages
- Years
- Common Crawl through August 2022
- Updated
- One-time release
- Size
- Size category above 1 TB; token count not in the record
Quality
Rare example of manual language audits at scale, with a paper describing the method.
- Snapshot ends in 2022
- Web text with unclear rights
- Audits cover samples, not every document
Connects to
- CulturaX: Alternative multilingual corpus drawn from mC4 and OSCAR
- FineWeb2: Newer multilingual corpus using fresher crawls
Questions it could answer
- Which languages have the largest clean versus noisy gap?
- How well do language labels hold up in a random audit?