Dolma
Allen Institute for AI
The three-trillion-token corpus behind OLMo, with the recipe and tooling published alongside it.
A mix of web content, academic papers, code, books, and encyclopedic text assembled to train the OLMo models. The Hub repository lists file URLs per version rather than a single table. A 10 billion token sample (v1_6-sample) exists for exploration.
Access
- Cost
- FreeNo charge; downloaded from file lists in the repository.
- License
- Open licenseODC-By 1.0 (changed from an earlier research license in April 2024)
- Sign-up
- No sign-up
- How you get it
- Bulk downloadJSON Lines (gzip)
What's in it
- One row is
- One row per document, with source and text
- Covers
- Global web, English
- Years
- Sources through 2023
- Updated
- Irregular
- Size
- About 3 trillion tokens; v1_5 is about 6.4 TB
Quality
Unusually transparent about construction, with open tooling for reproducing and inspecting it. Large and uneven by source.
- Files are served by URL list, not the datasets library's standard loader
- Contains web text with unclear rights for individual pages
Connects to
- Common Crawl: Source of most of the web portion
- The Pile: Earlier open mixed-source corpus that Dolma was built to improve on
- FineWeb: Larger web-only alternative often compared in ablations
Questions it could answer
- How does source mix change token counts across versions?
- What fraction of the corpus is code versus academic text?