The Pile
EleutherAI
The 825 GB open training set that taught the field how LLM data is assembled, and got sued over it.
A 2020 mix of 22 text sources for training language models, including web pages, PubMed and arXiv papers, GitHub code, Stack Exchange, books, and chat logs, with documentation of each component. It trained many early open models. Its Books3 component, built from pirated books, drew copyright takedowns and lawsuits, so the original release is no longer distributed as it was.
Access
- Cost
- FreeOriginally free download; full original copies are no longer officially hosted.
- License
- License not statedNot stated on the landing page; components carry their own terms
- Sign-up
- No sign-up
- How you get it
- Through the provider's siteJSON Lines (zstandard)
What's in it
- One row is
- One document from one of 22 named source subsets
- Covers
- Mostly English
- Years
- Released 2020
- Updated
- One-time release
- Size
- 825 GiB of text
Quality
Unusually well documented for its time, but its legal status is contested and the official full release is gone.
- Books3 was removed after copyright takedowns and is the subject of lawsuits
- Copies circulating online may differ from the original
- Mixes sources of very different quality and licensing
Connects to
- FineWeb: A newer, web-only open training set to compare with the Pile's curated mix
- Stack Exchange Data Dump: One of the Pile's 22 components
- Project Gutenberg: The Pile's PG-19 component comes from Gutenberg books
Questions it could answer
- What share of the Pile came from academic versus web versus code sources?
- Which components were most duplicated with each other?