Loghub: system log datasets for AI-driven log analytics
LOGPAI (Zhu, He, et al.)
Real system logs from supercomputers, Hadoop, Android, and more, for log-parsing research.
A collection of system log datasets from distributed systems, supercomputers, operating systems, mobile systems, and server applications. Some sets come with labeled anomalies or parsed templates. It is a standard benchmark for log parsing and anomaly detection research.
Access
- Cost
- FreeFree download from Zenodo; no account needed.
- License
- Open licenseCC BY 4.0 with a requirement to cite the Loghub paper and repository
- Sign-up
- No sign-up
- How you get it
- Bulk downloadGZ, ZIP
What's in it
- One row is
- One line (or session) of a system log
- Covers
- Not geographic
- Years
- Logs from various years; ISSRE 2023 release
- Updated
- One-time release
- Size
- Multiple log sets (up to millions of lines each)
Quality
Heavily used research benchmark with labels for some sets; the maintainers also host a GitHub repository.
- Logs are from specific systems and old software versions
- Anomaly labels are only available for some sets
Connects to
- HDFS log dataset: Anomaly-labeled Hadoop logs, one of the sets included
- BGL (Blue Gene/L) logs: Supercomputer logs, also included
- Loghub-2.0: Larger follow-up with more precise annotations
Questions it could answer
- Which parser recovers log templates most accurately across systems?
- Do anomaly detectors trained on one system's logs generalize to another?