What’s Out There
← Back to swiping
Internet & techFreeAcademic

Loghub: system log datasets for AI-driven log analytics

LOGPAI (Zhu, He, et al.)

Real system logs from supercomputers, Hadoop, Android, and more, for log-parsing research.

A collection of system log datasets from distributed systems, supercomputers, operating systems, mobile systems, and server applications. Some sets come with labeled anomalies or parsed templates. It is a standard benchmark for log parsing and anomaly detection research.

Access

Cost
FreeFree download from Zenodo; no account needed.
License
Open licenseCC BY 4.0 with a requirement to cite the Loghub paper and repository
Sign-up
No sign-up
How you get it
Bulk downloadGZ, ZIP

What's in it

One row is
One line (or session) of a system log
Covers
Not geographic
Years
Logs from various years; ISSRE 2023 release
Updated
One-time release
Size
Multiple log sets (up to millions of lines each)
timestamplog messagecomponentlog templateanomaly label

Quality

4 of 5

Heavily used research benchmark with labels for some sets; the maintainers also host a GitHub repository.

  • Logs are from specific systems and old software versions
  • Anomaly labels are only available for some sets

Connects to

  • HDFS log dataset: Anomaly-labeled Hadoop logs, one of the sets included
  • BGL (Blue Gene/L) logs: Supercomputer logs, also included
  • Loghub-2.0: Larger follow-up with more precise annotations

Questions it could answer

  • Which parser recovers log templates most accurately across systems?
  • Do anomaly detectors trained on one system's logs generalize to another?

Similar data sets

  • Waymo Open Dataset

    Lidar, camera, and motion logs from real robotaxis, released for anyone doing research.

    Images & videoWaymoFree

  • RedPajama-Data-V2

    Raw Common Crawl text in five languages with precomputed quality signals attached to each document.

    Language & textTogether AIFree

  • Jolpica F1 API (Ergast successor)

    Every Formula 1 race result since 1950, still queryable after the beloved Ergast API shut down.

    SportsJolpica (volunteer project)Free

  • LMSYS-Chat-1M

    One million real conversations with 25 LLMs, collected from Vicuna demo and Chatbot Arena visitors.

    Language & textLMSYS (UC Berkeley and collaborators)Free

  • Every Eval Ever Datastore

    One schema for AI evaluation results scraped from leaderboards, papers, and local runs, so they can be joined.

    Internet & techEvalEval Coalition (community)Free

  • Arena Leaderboard Dataset

    Historical snapshots of the Arena leaderboard, showing how model rankings moved over time.

    Internet & techLMArenaFree