What’s Out There
← Back to swiping
Language & textFreeNonprofit

Dolma

Allen Institute for AI

The three-trillion-token corpus behind OLMo, with the recipe and tooling published alongside it.

A mix of web content, academic papers, code, books, and encyclopedic text assembled to train the OLMo models. The Hub repository lists file URLs per version rather than a single table. A 10 billion token sample (v1_6-sample) exists for exploration.

Access

Cost
FreeNo charge; downloaded from file lists in the repository.
License
Open licenseODC-By 1.0 (changed from an earlier research license in April 2024)
Sign-up
No sign-up
How you get it
Bulk downloadJSON Lines (gzip)

What's in it

One row is
One row per document, with source and text
Covers
Global web, English
Years
Sources through 2023
Updated
Irregular
Size
About 3 trillion tokens; v1_5 is about 6.4 TB
textsourceidadded datemetadata

Quality

4 of 5

Unusually transparent about construction, with open tooling for reproducing and inspecting it. Large and uneven by source.

  • Files are served by URL list, not the datasets library's standard loader
  • Contains web text with unclear rights for individual pages

Connects to

  • Common Crawl: Source of most of the web portion
  • The Pile: Earlier open mixed-source corpus that Dolma was built to improve on
  • FineWeb: Larger web-only alternative often compared in ablations

Questions it could answer

  • How does source mix change token counts across versions?
  • What fraction of the corpus is code versus academic text?

Similar data sets

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free

  • Common Corpus

    2.27 trillion tokens of text that is either uncopyrighted or openly licensed, with per-document provenance.

    Language & textPleiasFree

  • The Pile

    The 825 GB open training set that taught the field how LLM data is assembled, and got sued over it.

    Language & textEleutherAIFree

  • OPUS Parallel Corpora

    Over 100 billion aligned sentence pairs across more than 1,000 languages.

    Language & textUniversity of Helsinki (Language Technology Research Group)Free

  • FineMath

    34 billion tokens of math-teaching web pages, chosen by a classifier trained on Llama annotations.

    EducationHugging Face (SmolLM team)Free

  • LMSYS-Chat-1M

    One million real conversations with 25 LLMs, collected from Vicuna demo and Chatbot Arena visitors.

    Language & textLMSYS (UC Berkeley and collaborators)Free