What’s Out There
← Back to swiping
Language & textPaidAcademic

English Gigaword Fifth Edition

Linguistic Data Consortium (University of Pennsylvania)

Ten billion words of newswire from seven agencies, the classic paid corpus behind early NLP.

Four million news articles from Agence France-Presse, the Associated Press, the New York Times, Xinhua, and three other English-language newswires, from 1994 to 2010. It was a standard training corpus for language models, summarization, and word embeddings before web-scale text, and it shows what sits behind a typical LDC license.

Access

Cost
PaidFee for non-members (shown after login); included for some LDC member institutions.
License
ProprietaryLDC User Agreement for Non-Members
Sign-up
Purchase required
How you get it
Bulk downloadSGML-tagged text

What's in it

One row is
One document per newswire story
Covers
International news from English-language newswires
Years
1994-2010
Updated
One-time release
Size
4,032,686 documents; about 9.9 billion tokens; 26 GB unzipped
document IDsource agencydateheadlinedatelinestory typebody text

Quality

4 of 5

Clean, consistently marked-up professional text with clear provenance; frozen in 2010.

  • No material after 2010
  • License restricts redistribution of the text
  • Newswire style is not representative of English generally

Connects to

Questions it could answer

  • How did Xinhua's English coverage of topics differ from AP's in the same years?
  • Which named entities rose fastest in newswire coverage between 1994 and 2010?

Similar data sets

  • Corpus of Contemporary American English (COCA)

    A billion words of American English, balanced across genres from TV scripts to academic journals.

    Language & textEnglish-Corpora.org (Mark Davies)Free tier, paid upgrades

  • Common Corpus

    2.27 trillion tokens of text that is either uncopyrighted or openly licensed, with per-document provenance.

    Language & textPleiasFree

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • XWord Info New York Times Crossword Archive

    Every New York Times crossword since the first on February 15, 1942, with grids, clues, and constructors.

    GamesXWord InfoFree tier, paid upgrades

  • xd Crossword Corpus

    Over 6 million crossword clue-answer pairs, from the project that caught a crossword plagiarist.

    Odditiesxd.saul.pw (Saul Pwanson)Free

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free