NCBI GenBank Complete (Parquet)
pulmo (individual), repackaging NCBI GenBank
GenBank's DNA sequences repackaged as Parquet for training genomic foundation models.
A processed copy of the NIH GenBank sequence database, in which each row is a nucleotide sequence with its accession. Large sequences are split into parts with suffixes. The card says it is redundant: sequences are included as submitted, unlike the curated RefSeq.
Access
- Cost
- FreeNo charge; very large.
- License
- Open licenseApache 2.0 (the repository's tag; NCBI's own terms govern the underlying data)
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One nucleotide sequence (or sequence chunk)
- Covers
- Global (INSDC member databases)
- Years
- Snapshot of GenBank (date on the Hub page)
- Updated
- One-time release
- Size
- Category above 1 trillion (the record lists n>1T)
Quality
Convenient for machine learning, but an individual's repackaging with only accession and sequence fields and no annotations.
- Individual maintainer, not NCBI
- Sequence annotations and metadata are dropped
- Redundant sequences, unlike RefSeq
Connects to
- GenBank: The original NCBI database with full annotations
- RefSeq: Curated, non-redundant reference sequences that avoid GenBank's redundancy
Questions it could answer
- How redundant are sequences across species?
- What share of sequence length comes from a few organisms?