HTTP Archive
HTTP Archive (community project sponsored by Google and others)
Every month, millions of websites crawled to record exactly how the web is built.
Monthly crawls of millions of desktop and mobile home pages (the URL list comes from Chrome's user experience report) run through WebPageTest, recording every request, byte, script, and third-party service, plus Lighthouse scores and detected technologies. It is the data behind the annual Web Almanac and most claims about page weight and web technology adoption.
Access
- Cost
- FreeFree public BigQuery dataset; you pay for queries beyond Google's free tier.
- License
- License not statedNot stated on the FAQ page
- Sign-up
- Free account
- How you get it
- Through the provider's siteBigQuery
What's in it
- One row is
- One row per page per crawl, with one row per HTTP request in the requests table
- Covers
- Global websites
- Years
- 2010-present
- Updated
- Monthly
- Size
- Millions of pages per monthly crawl; many terabytes in BigQuery
Quality
Consistent methodology, well documented, and maintained by experts; measures home pages from lab conditions, not real user journeys.
- Mostly home pages, not inner pages
- Test environment changes affect trends; check the changelog
- Full-table queries are costly
Connects to
- Common Crawl: Page content to pair with HTTP Archive's measurements
- Internet Archive Wayback Machine CDX API: See archived versions of pages HTTP Archive measured
Questions it could answer
- How much has median mobile page weight grown since 2015?
- What share of sites load a given analytics or ad tag?