Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis
This addresses a fundamental representational flaw in deep learning for vision tasks, offering both performance gains and efficiency improvements.
The paper tackles the problem of disordered weight structures in deep neural networks by introducing the Eigen Neural Network (ENN), which reparameterizes weights in a learned orthonormal eigenbasis to enforce decorrelated weight dynamics. This approach consistently outperforms state-of-the-art methods on ImageNet, sets a new benchmark in cross-modal image-text retrieval, and enables a backpropagation-free variant that achieves over 2× training speedup while surpassing end-to-end backpropagation accuracy.
The remarkable success of Deep Neural Networks(DNN) is driven by gradient-based optimization, yet this process is often undermined by its tendency to produce disordered weight structures, which harms feature clarity and degrades learning dynamics. To address this fundamental representational flaw, we introduced the Eigen Neural Network (ENN), a novel architecture that reparameterizes each layer's weights in a layer-shared, learned orthonormal eigenbasis. This design enforces decorrelated, well-aligned weight dynamics axiomatically, rather than through regularization, leading to more structured and discriminative feature representations. When integrated with standard BP, ENN consistently outperforms state-of-the-art methods on large-scale image classification benchmarks, including ImageNet, and its superior representations generalize to set a new benchmark in cross-modal image-text retrieval. Furthermore, ENN's principled structure enables a highly efficient, backpropagation-free(BP-free) local learning variant, ENN-$\ell$. This variant not only resolves BP's procedural bottlenecks to achieve over 2$\times$ training speedup via parallelism, but also, remarkably, surpasses the accuracy of end-to-end backpropagation. ENN thus presents a new architectural paradigm that directly remedies the representational deficiencies of BP, leading to enhanced performance and enabling a more efficient, parallelizable training regime.