What’s Out There
← Back to swiping
Internet & techFreeCompany

The Stack v2

BigCode (Hugging Face and ServiceNow)

Source code in more than 600 programming languages, with provenance tied to Software Heritage archives.

A large collection of permissively licensed and unlicensed source code used to train StarCoder2. The Hub holds file identifiers and metadata that point to the Software Heritage archive, with repository-grouped training variants. Access is gated and requires accepting terms, including an opt-out process for developers.

Access

Cost
FreeFree but gated; accept terms on the Hub.
License
Provider's own termsOther: per-file open-source licenses plus dataset terms of use
Sign-up
Free account
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per source file or repository (depending on variant)
Covers
Global open-source software
Years
Software Heritage archive snapshot (date not in record)
Updated
One-time release
Size
Billions of files across 600+ languages; exact total not in the record
file IDrepositorylanguagelicenseblob IDdetected license

Quality

4 of 5

Carefully documented provenance, license detection, and a developer opt-out process. File contents are fetched separately from Software Heritage.

  • Contents require a separate download from the archive
  • Mixed per-file licenses and attribution requirements
  • Contains secrets and personal data despite scrubbing efforts

Connects to

  • The Stack v3: Newer release from the same effort
  • StarCoderData: Cleaned training subset derived from the first Stack
  • GitHub Code: Smaller GitHub-only code corpus

Questions it could answer

  • Which languages dominate by file count versus by bytes?
  • What share of files carries permissive licenses?

Similar data sets

  • Stack Exchange Data Dump

    Every public question, answer, comment, and vote on Stack Overflow and 170+ sister sites.

    Internet & techStack Exchange, Inc.Free

  • The Pile

    The 825 GB open training set that taught the field how LLM data is assembled, and got sued over it.

    Language & textEleutherAIFree

  • Glottolog

    A stable ID and family tree for every known language and dialect, plus 471,000 works describing them.

    Arts & cultureMax Planck Institute for Evolutionary AnthropologyFree

  • GH Archive

    Every public push, star, fork, and issue comment on GitHub, archived hour by hour since 2011.

    Internet & techGH Archive (Ilya Grigorik)Free

  • Project Gutenberg

    Tens of thousands of public-domain books, hand-proofed by volunteers since 1971.

    Language & textProject Gutenberg Literary Archive FoundationFree

  • Hacker News Complete Archive

    Every Hacker News story, comment, and poll since 2006, refreshed every five minutes.

    Social mediaopen-indexFree