Paths and Ambient Spaces in Neural Loss Landscapes
This work addresses the challenge of analyzing loss landscapes for neural network theory and practice, with incremental improvements in Bayesian neural network inference.
The paper tackles the problem of understanding neural network loss surfaces by proposing a method to embed loss tunnels, revealing insights into their length and structure and correcting misconceptions. It applies this to Bayesian neural networks, improving subspace inference by identifying pitfalls and proposing a more natural prior for better sampling.
Understanding the structure of neural network loss surfaces, particularly the emergence of low-loss tunnels, is critical for advancing neural network theory and practice. In this paper, we propose a novel approach to directly embed loss tunnels into the loss landscape of neural networks. Exploring the properties of these loss tunnels offers new insights into their length and structure and sheds light on some common misconceptions. We then apply our approach to Bayesian neural networks, where we improve subspace inference by identifying pitfalls and proposing a more natural prior that better guides the sampling procedure.