Internet Archive Wayback Machine CDX API
Internet Archive
A queryable index of over a trillion web page captures going back to 1996.
The CDX server returns the list of Wayback Machine captures for a URL or domain: when each snapshot was taken, its HTTP status, content type, and content digest. It lets analysts reconstruct how a site changed, find when pages appeared or disappeared, and pull historical versions for research without crawling the live web.
Access
- Cost
- FreeFree, no key; heavy users are rate limited.
- License
- Provider's own termsInternet Archive terms of use; archived content keeps its owners' copyrights
- Sign-up
- No sign-up
- How you get it
- APIPlain text, JSON
What's in it
- One row is
- One row per archived capture of a URL
- Covers
- Global web
- Years
- 1996-present
- Updated
- Real time
- Size
- Over a trillion archived web pages
Quality
Irreplaceable historical record with a simple, stable API; capture frequency is uneven and some sites are excluded.
- Popular sites are captured far more often than obscure ones
- Site owners can request exclusion
- Rate limits slow large studies
Connects to
- Common Crawl: Fill gaps in Wayback coverage with Common Crawl snapshots, and vice versa
- HTTP Archive: Measurements of how pages were built, to pair with archived copies
Questions it could answer
- When did a set of government web pages disappear?
- How often were major news homepages captured each year?