English Gigaword Fifth Edition
Linguistic Data Consortium (University of Pennsylvania)
Ten billion words of newswire from seven agencies, the classic paid corpus behind early NLP.
Four million news articles from Agence France-Presse, the Associated Press, the New York Times, Xinhua, and three other English-language newswires, from 1994 to 2010. It was a standard training corpus for language models, summarization, and word embeddings before web-scale text, and it shows what sits behind a typical LDC license.
Access
- Cost
- PaidFee for non-members (shown after login); included for some LDC member institutions.
- License
- ProprietaryLDC User Agreement for Non-Members
- Sign-up
- Purchase required
- How you get it
- Bulk downloadSGML-tagged text
What's in it
- One row is
- One document per newswire story
- Covers
- International news from English-language newswires
- Years
- 1994-2010
- Updated
- One-time release
- Size
- 4,032,686 documents; about 9.9 billion tokens; 26 GB unzipped
Quality
Clean, consistently marked-up professional text with clear provenance; frozen in 2010.
- No material after 2010
- License restricts redistribution of the text
- Newswire style is not representative of English generally
Connects to
- Corpus of Contemporary American English (COCA): A genre-balanced corpus to set newswire language against
- GDELT Event Database and Global Knowledge Graph: GDELT covers later news years with coded events rather than full text
Questions it could answer
- How did Xinhua's English coverage of topics differ from AP's in the same years?
- Which named entities rose fastest in newswire coverage between 1994 and 2010?