What’s Out There
← Back to swiping
Language & textFreeNonprofit

Common Crawl

Common Crawl Foundation

A free, petabyte-scale copy of the public web, re-crawled again and again since 2008.

Raw web pages, metadata, and extracted plain text from billions of URLs per crawl, published as numbered snapshots (CC-MAIN-YYYY-WW). It is the base layer under most large language model training sets, and it also comes with URL indexes and host- and domain-level web graphs.

Access

Cost
FreeFree to download or query in place on AWS; you pay your own compute and egress.
License
Provider's own termsCommon Crawl Terms of Use; underlying pages keep their own copyrights
Sign-up
No sign-up
How you get it
API and Bulk downloadWARC, WAT, WET, CDXJ index, Parquet index

What's in it

One row is
One record per fetched web page (HTTP response), grouped into crawl snapshots
Covers
Global public web, weighted toward English-language sites
Years
2008-present
Updated
Monthly
Size
Petabytes in total; each snapshot covers billions of pages
URLfetch timestampHTTP headersraw HTMLextracted textoutbound linksdetected language

Quality

4 of 5

Long-running, consistently formatted, and heavily used; but it is a crawl, not a census, and contains whatever junk the web contains.

  • Coverage of any one site is partial and varies between snapshots
  • Contains spam, duplicates, and copyrighted or personal content
  • Robots.txt opt-outs have grown since 2023, especially among news sites

Connects to

  • FineWeb: A filtered, deduplicated English text set built from Common Crawl snapshots
  • Internet Archive Wayback Machine CDX API: Check whether a page Common Crawl missed was captured by the Wayback Machine
  • HTTP Archive: Pairs Common Crawl's content with HTTP Archive's measurements of how pages are built

Questions it could answer

  • How has the share of web pages in each language changed across crawls since 2013?
  • Which domains are most linked-to in the latest host-level web graph?
  • How many news sites now block the CCBot crawler?

Similar data sets

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • Internet Archive Wayback Machine CDX API

    A queryable index of over a trillion web page captures going back to 1996.

    Internet & techInternet ArchiveFree

  • MADLAD-400

    A Common Crawl corpus in 419 languages whose lower-resource portions were manually audited.

    Language & textGoogle Research and Allen Institute for AIFree

  • DCLM-baseline

    A 4 trillion token pretraining set that won a controlled contest of data filtering recipes.

    Language & textDataComp for Language Models (mlfoundations)Free

  • HTTP Archive

    Every month, millions of websites crawled to record exactly how the web is built.

    Internet & techHTTP Archive (community project sponsored by Google and others)Free

  • RedPajama-Data-V2

    Raw Common Crawl text in five languages with precomputed quality signals attached to each document.

    Language & textTogether AIFree