What’s Out There
← Back to swiping
Internet & techFreeIndividual

GH Archive

GH Archive (Ilya Grigorik)

Every public push, star, fork, and issue comment on GitHub, archived hour by hour since 2011.

Hourly files of the public GitHub event stream (15+ event types, from pushes and pull requests to stars and new members) as reported by GitHub's API, also loaded into a public BigQuery dataset. Researchers use it to measure open-source activity, project popularity, developer behavior, and the spread of tools and languages.

Access

Cost
FreeFree file downloads; BigQuery queries are billed by Google after its free tier.
License
License not statedNot stated; events come from GitHub's public API
Sign-up
No sign-up
How you get it
Bulk downloadJSON (gzip), BigQuery

What's in it

One row is
One row per public GitHub event
Covers
Global (GitHub users)
Years
February 2011-present
Updated
Real time
Size
Billions of events; one gzipped JSON file per hour
event typeactorrepositorycreated atpayloadorganization

Quality

4 of 5

Complete for public events and easy to use; event formats changed when GitHub's API did.

  • 2011-2014 data came from a deprecated Timeline API with a different schema
  • Private repository activity is absent
  • Bot accounts generate a large share of events

Connects to

Questions it could answer

  • Which programming languages gained the most new repositories each year?
  • How long do pull requests take to merge in large versus small projects?
  • What share of GitHub events now come from bots?

Similar data sets

  • Hacker News Complete Archive

    Every Hacker News story, comment, and poll since 2006, refreshed every five minutes.

    Social mediaopen-indexFree

  • Archive Team Twitter Stream Grab

    Twelve years of Twitter's free 1% sample stream, saved month by month until the tap closed.

    Social mediaArchive Team (hosted by the Internet Archive)Free

  • HTTP Archive

    Every month, millions of websites crawled to record exactly how the web is built.

    Internet & techHTTP Archive (community project sponsored by Google and others)Free

  • PyPI Download Statistics (BigQuery)

    Every pip install of every Python package, logged and queryable in SQL.

    Internet & techPython Software Foundation (PyPI)Free

  • The Stack v2

    Source code in more than 600 programming languages, with provenance tied to Software Heritage archives.

    Internet & techBigCode (Hugging Face and ServiceNow)Free

  • Stack Exchange Data Dump

    Every public question, answer, comment, and vote on Stack Overflow and 170+ sister sites.

    Internet & techStack Exchange, Inc.Free