Develop End-to-End Anomaly Detection System
This work addresses the problem of developing robust anomaly detection systems for computer networking, where data limitations and complex anomaly patterns hinder practical implementation, though it appears incremental as it builds on existing methods with a new pipeline and model.
The paper tackles the challenge of anomaly detection in computer networks where anomalies can have multiple causes and labeled data is scarce, by proposing an end-to-end development pipeline that incorporates user feedback for continuous evaluation and optimization. It demonstrates the effectiveness of this approach by benchmarking a new forecasting model called Lachesis on a real-world networking problem, showing robustness compared to existing models.
Anomaly detection plays a crucial role in ensuring network robustness. However, implementing intelligent alerting systems becomes a challenge when considering scenarios in which anomalies can be caused by both malicious and non-malicious events, leading to the difficulty of determining anomaly patterns. The lack of labeled data in the computer networking domain further exacerbates this issue, impeding the development of robust models capable of handling real-world scenarios. To address this challenge, in this paper, we propose an end-to-end anomaly detection model development pipeline. This framework makes it possible to consume user feedback and enable continuous user-centric model performance evaluation and optimization. We demonstrate the efficacy of the framework by way of introducing and bench-marking a new forecasting model -- named \emph{Lachesis} -- on a real-world networking problem. Experiments have demonstrated the robustness and effectiveness of the two proposed versions of \emph{Lachesis} compared with other models proposed in the literature. Our findings underscore the potential for improving the performance of data-driven products over their life cycles through a harmonized integration of user feedback and iterative development.