What’s Out There
← Back to swiping
Internet & techFreeCommunity project

HTTP Archive

HTTP Archive (community project sponsored by Google and others)

Every month, millions of websites crawled to record exactly how the web is built.

Monthly crawls of millions of desktop and mobile home pages (the URL list comes from Chrome's user experience report) run through WebPageTest, recording every request, byte, script, and third-party service, plus Lighthouse scores and detected technologies. It is the data behind the annual Web Almanac and most claims about page weight and web technology adoption.

Access

Cost
FreeFree public BigQuery dataset; you pay for queries beyond Google's free tier.
License
License not statedNot stated on the FAQ page
Sign-up
Free account
How you get it
Through the provider's siteBigQuery

What's in it

One row is
One row per page per crawl, with one row per HTTP request in the requests table
Covers
Global websites
Years
2010-present
Updated
Monthly
Size
Millions of pages per monthly crawl; many terabytes in BigQuery
page URLcrawl dateclient (desktop or mobile)total bytesrequest counttechnologies detectedLighthouse scores

Quality

4 of 5

Consistent methodology, well documented, and maintained by experts; measures home pages from lab conditions, not real user journeys.

  • Mostly home pages, not inner pages
  • Test environment changes affect trends; check the changelog
  • Full-table queries are costly

Connects to

Questions it could answer

  • How much has median mobile page weight grown since 2015?
  • What share of sites load a given analytics or ad tag?

Similar data sets

  • Internet Archive Wayback Machine CDX API

    A queryable index of over a trillion web page captures going back to 1996.

    Internet & techInternet ArchiveFree

  • Common Crawl

    A free, petabyte-scale copy of the public web, re-crawled again and again since 2008.

    Language & textCommon Crawl FoundationFree

  • Cloudflare Radar

    Global internet traffic, outages, attacks, and bot trends, seen from one of the web's largest networks.

    Internet & techCloudflareFree

  • GH Archive

    Every public push, star, fork, and issue comment on GitHub, archived hour by hour since 2011.

    Internet & techGH Archive (Ilya Grigorik)Free

  • CAIDA Anonymized Internet Traces

    Packet-level traffic from commercial internet backbone links, the rawest view of the internet on offer.

    Internet & techCAIDA (Center for Applied Internet Data Analysis, UC San Diego)By application

  • Archive Team Twitter Stream Grab

    Twelve years of Twitter's free 1% sample stream, saved month by month until the tap closed.

    Social mediaArchive Team (hosted by the Internet Archive)Free