The Stack v2
BigCode (Hugging Face and ServiceNow)
Source code in more than 600 programming languages, with provenance tied to Software Heritage archives.
A large collection of permissively licensed and unlicensed source code used to train StarCoder2. The Hub holds file identifiers and metadata that point to the Software Heritage archive, with repository-grouped training variants. Access is gated and requires accepting terms, including an opt-out process for developers.
Access
- Cost
- FreeFree but gated; accept terms on the Hub.
- License
- Provider's own termsOther: per-file open-source licenses plus dataset terms of use
- Sign-up
- Free account
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per source file or repository (depending on variant)
- Covers
- Global open-source software
- Years
- Software Heritage archive snapshot (date not in record)
- Updated
- One-time release
- Size
- Billions of files across 600+ languages; exact total not in the record
Quality
Carefully documented provenance, license detection, and a developer opt-out process. File contents are fetched separately from Software Heritage.
- Contents require a separate download from the archive
- Mixed per-file licenses and attribution requirements
- Contains secrets and personal data despite scrubbing efforts
Connects to
- The Stack v3: Newer release from the same effort
- StarCoderData: Cleaned training subset derived from the first Stack
- GitHub Code: Smaller GitHub-only code corpus
Questions it could answer
- Which languages dominate by file count versus by bytes?
- What share of files carries permissive licenses?