Common Crawl
Common Crawl Foundation
A free, petabyte-scale copy of the public web, re-crawled again and again since 2008.
Raw web pages, metadata, and extracted plain text from billions of URLs per crawl, published as numbered snapshots (CC-MAIN-YYYY-WW). It is the base layer under most large language model training sets, and it also comes with URL indexes and host- and domain-level web graphs.
Access
- Cost
- FreeFree to download or query in place on AWS; you pay your own compute and egress.
- License
- Provider's own termsCommon Crawl Terms of Use; underlying pages keep their own copyrights
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadWARC, WAT, WET, CDXJ index, Parquet index
What's in it
- One row is
- One record per fetched web page (HTTP response), grouped into crawl snapshots
- Covers
- Global public web, weighted toward English-language sites
- Years
- 2008-present
- Updated
- Monthly
- Size
- Petabytes in total; each snapshot covers billions of pages
Quality
Long-running, consistently formatted, and heavily used; but it is a crawl, not a census, and contains whatever junk the web contains.
- Coverage of any one site is partial and varies between snapshots
- Contains spam, duplicates, and copyrighted or personal content
- Robots.txt opt-outs have grown since 2023, especially among news sites
Connects to
- FineWeb: A filtered, deduplicated English text set built from Common Crawl snapshots
- Internet Archive Wayback Machine CDX API: Check whether a page Common Crawl missed was captured by the Wayback Machine
- HTTP Archive: Pairs Common Crawl's content with HTTP Archive's measurements of how pages are built
Questions it could answer
- How has the share of web pages in each language changed across crawls since 2013?
- Which domains are most linked-to in the latest host-level web graph?
- How many news sites now block the CCBot crawler?