VAR: Visual Analysis for Rashomon Set of Machine Learning Models' Performance
This work addresses a visualization bottleneck for machine learning model developers analyzing Rashomon sets, but it is incremental as it builds on existing concepts with a new tool.
The paper tackles the problem of comparing multiple closely matched machine learning models in the Rashomon set, which lack effective visualization methods for horizontal feature-based analysis, and proposes VAR, a visualization solution using heatmaps and scatter plots to help developers identify optimal models and understand set characteristics.
Evaluating the performance of closely matched machine learning(ML) models under specific conditions has long been a focus of researchers in the field of machine learning. The Rashomon set is a collection of closely matched ML models, encompassing a wide range of models with similar accuracies but different structures. Traditionally, the analysis of these sets has focused on vertical structural analysis, which involves comparing the corresponding features at various levels within the ML models. However, there has been a lack of effective visualization methods for horizontally comparing multiple models with specific features. We propose the VAR visualization solution. VAR uses visualization to perform comparisons of ML models within the Rashomon set. This solution combines heatmaps and scatter plots to facilitate the comparison. With the help of VAR, ML model developers can identify the optimal model under specific conditions and better understand the Rashomon set's overall characteristics.