Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs
This provides a practical tool for researchers and practitioners in deep learning and optimization to assess the stability of multi-objective methods, though it is incremental as it implements an existing concept.
The authors tackled the problem of evaluating performance stability in multi-objective optimization by developing a Python package for visualizing the variability of Pareto fronts over multiple runs, addressing the lack of existing tools for empirical attainment surfaces.
Hyperparameter optimization is crucial to achieving high performance in deep learning. On top of the performance, other criteria such as inference time or memory requirement often need to be optimized due to some practical reasons. This motivates research on multi-objective optimization (MOO). However, Pareto fronts of MOO methods are often shown without considering the variability caused by random seeds and this makes the performance stability evaluation difficult. Although there is a concept named empirical attainment surface to enable the visualization with uncertainty over multiple runs, there is no major Python package for empirical attainment surface. We, therefore, develop a Python package for this purpose and describe the usage. The package is available at https://github.com/nabenabe0928/empirical-attainment-func.