What’s Out There
← Back to swiping
Language & textFreeCompany

FineWeb-Edu

Hugging Face (FineWeb team)

1.3 trillion tokens of web text kept because a classifier judged it educational.

A subset of FineWeb filtered with a classifier trained on Llama-3-70B-Instruct annotations of educational quality. One row is one web page with its text, URL, and an educational score. The card also describes a larger 5.4 trillion token version with a lower score cutoff.

Access

Cost
FreeNo charge; Parquet files on the Hub.
License
Open licenseODC-By 1.0
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per web document
Covers
Global web, English only
Years
Common Crawl snapshots through 2024
Updated
Irregular
Size
1.3 trillion tokens (5.4 trillion in the lower-threshold variant)
texturleducational scorecrawl dump IDtoken count

Quality

4 of 5

Documented in a paper, and models trained on it score well on knowledge benchmarks for their size. The filter reflects an LLM's idea of educational content.

  • Quality labels come from an LLM classifier, so its biases carry through
  • Heavy tilt toward school-style explanatory prose
  • Benchmark questions may leak because the classifier favors textbook-like pages

Connects to

  • FineWeb: Parent corpus this was filtered from
  • SmolLM-Corpus: Includes FineWeb-Edu as one of its three subsets

Questions it could answer

  • Which domains contribute most to the educational subset compared with the full FineWeb?
  • How does the educational score distribute across crawl years?

Similar data sets

  • FineWeb

    Trillions of tokens of cleaned English web text, with every filtering step published.

    Language & textHugging FaceFree

  • FineWeb2

    Pretraining text for over 1,000 languages, with the pipeline tuned and validated language by language.

    Language & textHugging Face (FineWeb team)Free

  • FineMath

    34 billion tokens of math-teaching web pages, chosen by a classifier trained on Llama annotations.

    EducationHugging Face (SmolLM team)Free

  • FinePDFs

    About 3 trillion tokens extracted from PDFs, a source most web corpora skip because extraction is costly.

    Language & textHugging Face (FineWeb team)Free

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free

  • DCLM-baseline

    A 4 trillion token pretraining set that won a controlled contest of data filtering recipes.

    Language & textDataComp for Language Models (mlfoundations)Free