eACGM: Non-instrumented Performance Tracing and Anomaly Detection towards Machine Learning Systems
This addresses performance optimization and fault diagnosis for large-scale AI/ML systems, but it is incremental as it builds on existing monitoring and anomaly detection techniques.
The paper tackles the problem of monitoring AI/ML systems by introducing eACGM, a non-instrumented framework that collects real-time performance data from hardware and software components using eBPF and applies a Gaussian Mixture Model for anomaly detection, with results showing it successfully captures critical anomalies while maintaining low overhead in distributed training scenarios.
We present eACGM, a full-stack AI/ML system monitoring framework based on eBPF. eACGM collects real-time performance data from key hardware components, including the GPU and network communication layer, as well as from key software stacks such as CUDA, Python, and PyTorch, all without requiring any code instrumentation or modifications. Additionally, it leverages libnvml to gather process-level GPU resource usage information. By applying a Gaussian Mixture Model (GMM) to the collected multidimensional performance metrics for statistical modeling and clustering analysis, eACGM effectively identifies complex failure modes, such as latency anomalies, hardware failures, and communication inefficiencies, enabling rapid diagnosis of system bottlenecks and abnormal behaviors. To evaluate eACGM's effectiveness and practicality, we conducted extensive empirical studies and case analyses in multi-node distributed training scenarios. The results demonstrate that eACGM, while maintaining a non-intrusive and low-overhead profile, successfully captures critical performance anomalies during model training and inference. Its stable anomaly detection performance and comprehensive monitoring capabilities validate its applicability and scalability in real-world production environments, providing strong support for performance optimization and fault diagnosis in large-scale AI/ML systems.