What’s Out There
← Back to swiping
Internet & techFreeNonprofit

Internet Archive Wayback Machine CDX API

Internet Archive

A queryable index of over a trillion web page captures going back to 1996.

The CDX server returns the list of Wayback Machine captures for a URL or domain: when each snapshot was taken, its HTTP status, content type, and content digest. It lets analysts reconstruct how a site changed, find when pages appeared or disappeared, and pull historical versions for research without crawling the live web.

Access

Cost
FreeFree, no key; heavy users are rate limited.
License
Provider's own termsInternet Archive terms of use; archived content keeps its owners' copyrights
Sign-up
No sign-up
How you get it
APIPlain text, JSON

What's in it

One row is
One row per archived capture of a URL
Covers
Global web
Years
1996-present
Updated
Real time
Size
Over a trillion archived web pages
URL keytimestamporiginal URLMIME typeHTTP statusdigestlength

Quality

4 of 5

Irreplaceable historical record with a simple, stable API; capture frequency is uneven and some sites are excluded.

  • Popular sites are captured far more often than obscure ones
  • Site owners can request exclusion
  • Rate limits slow large studies

Connects to

  • Common Crawl: Fill gaps in Wayback coverage with Common Crawl snapshots, and vice versa
  • HTTP Archive: Measurements of how pages were built, to pair with archived copies

Questions it could answer

  • When did a set of government web pages disappear?
  • How often were major news homepages captured each year?

Similar data sets

  • HTTP Archive

    Every month, millions of websites crawled to record exactly how the web is built.

    Internet & techHTTP Archive (community project sponsored by Google and others)Free

  • Common Crawl

    A free, petabyte-scale copy of the public web, re-crawled again and again since 2008.

    Language & textCommon Crawl FoundationFree

  • Archive Team Twitter Stream Grab

    Twelve years of Twitter's free 1% sample stream, saved month by month until the tap closed.

    Social mediaArchive Team (hosted by the Internet Archive)Free

  • Wikipedia Database Dumps

    Every article and every past revision of every Wikipedia, free to download.

    Language & textWikimedia FoundationFree

  • X (Twitter) API

    Once free for academics, Twitter data now costs credits per post read.

    Social mediaX Corp.Paid

  • Hacker News Complete Archive

    Every Hacker News story, comment, and poll since 2006, refreshed every five minutes.

    Social mediaopen-indexFree