What’s Out There
← Back to swiping
Language & textFreeAcademic

OPUS Parallel Corpora

University of Helsinki (Language Technology Research Group)

Over 100 billion aligned sentence pairs across more than 1,000 languages.

A collection of 1,200+ sentence-aligned translation corpora gathered from subtitles, EU and UN documents, software localization files, religious texts, and web crawls. Each pair links a sentence to its translation, which makes OPUS the first stop for training or evaluating machine translation, especially for smaller languages.

Access

Cost
FreeFree downloads and APIs.
License
Provider's own termsVaries by corpus; each corpus page states its source's terms
Sign-up
No sign-up
How you get it
API and Bulk downloadMoses plain text, TMX, XML

What's in it

One row is
One row per aligned sentence pair within a corpus and language pair
Covers
1,038 languages
Years
Varies by corpus
Updated
Irregular
Size
About 102.9 billion sentence pairs across 1,214 corpora
source sentencetarget sentencelanguage paircorpus namedocument IDalignment score

Quality

4 of 5

The most complete hub for parallel text, though quality ranges from professional translation to noisy web-mined pairs.

  • OpenSubtitles and NLLB make up nearly half the pairs and are noisy
  • Licenses differ corpus by corpus
  • Low-resource pairs are often mostly religious or software text

Connects to

Questions it could answer

  • For which languages does nearly all available parallel text come from Bible translations?
  • How much aligned text links each African language to English versus French?

Similar data sets

  • Multilingual parallel text corpora for East African languages

    English text translated into five East African languages, for training machine translation.

    Language & textMakerere University NLP groupFree

  • Common Corpus

    2.27 trillion tokens of text that is either uncopyrighted or openly licensed, with per-document provenance.

    Language & textPleiasFree

  • Dolma

    The three-trillion-token corpus behind OLMo, with the recipe and tooling published alongside it.

    Language & textAllen Institute for AIFree

  • OpenAssistant Conversations (OASST1)

    161,443 chat messages written and ranked by volunteers in 35 languages, with no model in the loop.

    Language & textOpenAssistant (LAION-affiliated volunteers)Free

  • MultiCoNER Datasets

    Multilingual named-entity recognition data with hard cases like movie titles and short text.

    Language & textAmazonFree

  • xd Crossword Corpus

    Over 6 million crossword clue-answer pairs, from the project that caught a crossword plagiarist.

    Odditiesxd.saul.pw (Saul Pwanson)Free