KPIs-Based Clustering and Visualization of HPC jobs: a Feature Reduction Approach
This work addresses the need for efficient monitoring and analysis of HPC systems to improve management and issue detection, but it is incremental as it applies existing feature reduction and clustering methods to a specific domain.
The paper tackles the problem of analyzing high-dimensional monitoring data from High-Performance Computing (HPC) systems by introducing a methodology to cluster HPC jobs based on Key Performance Indicators (KPIs), using feature reduction techniques like literature-based and variance-based extraction, and validating it on real data to show that CPU-related KPIs yield the best clustering results.
High-Performance Computing (HPC) systems need to be constantly monitored to ensure their stability. The monitoring systems collect a tremendous amount of data about different parameters or Key Performance Indicators (KPIs), such as resource usage, IO waiting time, etc. A proper analysis of this data, usually stored as time series, can provide insight in choosing the right management strategies as well as the early detection of issues. In this paper, we introduce a methodology to cluster HPC jobs according to their KPI indicators. Our approach reduces the inherent high dimensionality of the collected data by applying two techniques to the time series: literature-based and variance-based feature extraction. We also define a procedure to visualize the obtained clusters by combining the two previous approaches and the Principal Component Analysis (PCA). Finally, we have validated our contributions on a real data set to conclude that those KPIs related to CPU usage provide the best cohesion and separation for clustering analysis and the good results of our visualization methodology.