What’s Out There
← Back to swiping
Language & textFreeNonprofit

The Pile

EleutherAI

The 825 GB open training set that taught the field how LLM data is assembled, and got sued over it.

A 2020 mix of 22 text sources for training language models, including web pages, PubMed and arXiv papers, GitHub code, Stack Exchange, books, and chat logs, with documentation of each component. It trained many early open models. Its Books3 component, built from pirated books, drew copyright takedowns and lawsuits, so the original release is no longer distributed as it was.

Access

Cost
FreeOriginally free download; full original copies are no longer officially hosted.
License
License not statedNot stated on the landing page; components carry their own terms
Sign-up
No sign-up
How you get it
Through the provider's siteJSON Lines (zstandard)

What's in it

One row is
One document from one of 22 named source subsets
Covers
Mostly English
Years
Released 2020
Updated
One-time release
Size
825 GiB of text
textsource subsetdocument metadatatrain/validation/test split

Quality

3 of 5

Unusually well documented for its time, but its legal status is contested and the official full release is gone.

  • Books3 was removed after copyright takedowns and is the subject of lawsuits
  • Copies circulating online may differ from the original
  • Mixes sources of very different quality and licensing

Connects to

Questions it could answer

  • What share of the Pile came from academic versus web versus code sources?
  • Which components were most duplicated with each other?

Similar data sets

  • Dolma

    The three-trillion-token corpus behind OLMo, with the recipe and tooling published alongside it.

    Language & textAllen Institute for AIFree

  • MMMLU (Multilingual MMLU)

    The MMLU test set translated into 14 languages by professional human translators.

    EducationOpenAIFree

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • Common Corpus

    2.27 trillion tokens of text that is either uncopyrighted or openly licensed, with per-document provenance.

    Language & textPleiasFree

  • LMSYS-Chat-1M

    One million real conversations with 25 LLMs, collected from Vicuna demo and Chatbot Arena visitors.

    Language & textLMSYS (UC Berkeley and collaborators)Free