FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks
This tool helps researchers better understand neural network behavior through visualization, though it is incremental as it builds on existing slicing methods.
The researchers tackled the problem of visualizing high-dimensional neural network loss landscapes by developing FuNNscope, an interactive visualization tool that uses interpretable axes to approximate loss geometry. They demonstrated the tool's effectiveness through observations of symmetries, weight sensitivities, and gradient descent navigation, achieving an average System Usability Scale score in user studies.
Despite their effective use in various fields, many aspects of neural networks are poorly understood. One important way to investigate the characteristics of neural networks is to explore the loss landscape. However, most models produce a high-dimensional non-convex landscape which is difficult to visualize. We discuss and extend existing visualization methods based on 1D- and 2D slicing with a novel method that approximates the actual loss landscape geometry by using charts with interpretable axes. Based on the assumption that observations on small neural networks can generalize to more complex systems and provide us with helpful insights, we focus on small models in the range of a few dozen weights, which enables computationally cheap experiments and the use of an interactive dashboard. We observe symmetries around the zero vector, the influence of different layers on the global landscape, the different weight sensitivities around a minimizer, and how gradient descent navigates high-loss obstacles. The user study resulted in an average SUS (System Usability Scale) score with suggestions for improvement and opened up a number of possible application scenarios, such as autoencoders and ensemble networks.