Spectral Neural Networks: Approximation Theory and Optimization Landscape
This work addresses theoretical gaps in SNNs, which are used as alternatives to traditional eigensolvers in machine learning for spectral geometric data analysis, but it is incremental as it builds on existing SNN approaches.
The paper investigates the theoretical aspects of Spectral Neural Networks (SNNs), analyzing the tradeoff between neuron count and learned spectral information and exploring the optimization landscape to understand training dynamics, with a focus on the challenges posed by non-convex loss functions.
There is a large variety of machine learning methodologies that are based on the extraction of spectral geometric information from data. However, the implementations of many of these methods often depend on traditional eigensolvers, which present limitations when applied in practical online big data scenarios. To address some of these challenges, researchers have proposed different strategies for training neural networks as alternatives to traditional eigensolvers, with one such approach known as Spectral Neural Network (SNN). In this paper, we investigate key theoretical aspects of SNN. First, we present quantitative insights into the tradeoff between the number of neurons and the amount of spectral geometric information a neural network learns. Second, we initiate a theoretical exploration of the optimization landscape of SNN's objective to shed light on the training dynamics of SNN. Unlike typical studies of convergence to global solutions of NN training dynamics, SNN presents an additional complexity due to its non-convex ambient loss function.