Dynamic Spectral Backpropagation for Efficient Neural Network Training
This work addresses efficiency and robustness challenges in neural network training for resource-limited applications, though it appears incremental as it builds on existing gradient and spectral methods.
The paper tackles efficient neural network training under resource constraints by introducing Dynamic Spectral Backpropagation (DSBP), which projects gradients onto principal eigenvectors to reduce complexity and promote flat minima, outperforming methods like SAM, LoRA, and MAML on datasets such as CIFAR-10, Fashion MNIST, MedMNIST, and Tiny ImageNet.
Dynamic Spectral Backpropagation (DSBP) enhances neural network training under resource constraints by projecting gradients onto principal eigenvectors, reducing complexity and promoting flat minima. Five extensions are proposed, dynamic spectral inference, spectral architecture optimization, spectral meta learning, spectral transfer regularization, and Lie algebra inspired dynamics, to address challenges in robustness, fewshot learning, and hardware efficiency. Supported by a third order stochastic differential equation (SDE) and a PAC Bayes limit, DSBP outperforms Sharpness Aware Minimization (SAM), Low Rank Adaptation (LoRA), and Model Agnostic Meta Learning (MAML) on CIFAR 10, Fashion MNIST, MedMNIST, and Tiny ImageNet, as demonstrated through extensive experiments and visualizations. Future work focuses on scalability, bias mitigation, and ethical considerations.