Linnaeus: A highly reusable and adaptable ML based log classification pipeline
This work provides a practical solution for software engineers and developers in industrial settings to manage and analyze logs efficiently, though it is incremental as it builds on existing ML methods.
The paper tackles the problem of automating log analysis in large-scale software systems by presenting Linnaeus, an end-to-end machine learning pipeline for log classification, which addresses practical aspects like adaptability, reusability, and integration into development processes.
Logs are a common way to record detailed run-time information in software. As modern software systems evolve in scale and complexity, logs have become indispensable to understanding the internal states of the system. At the same time however, manually inspecting logs has become impractical. In recent times, there has been more emphasis on statistical and machine learning (ML) based methods for analyzing logs. While the results have shown promise, most of the literature focuses on algorithms and state-of-the-art (SOTA), while largely ignoring the practical aspects. In this paper we demonstrate our end-to-end log classification pipeline, Linnaeus. Besides showing the more traditional ML flow, we also demonstrate our solutions for adaptability and re-use, integration towards large scale software development processes, and how we cope with lack of labelled data. We hope Linnaeus can serve as a blueprint for, and inspire the integration of, various ML based solutions in other large scale industrial settings.