SAL: Selective Adaptive Learning for Backpropagation-Free Training with Sparsification
This addresses the problem of scalable and biologically plausible training methods for deep learning researchers, though it appears incremental as it builds on feedback alignment and sparsification techniques.
The paper tackles the limitations of backpropagation in deep learning, such as biologically implausible weight symmetry and gradient interference, by proposing Selective Adaptive Learning (SAL), a method that combines selective parameter activation with adaptive area partitioning, resulting in competitive convergence rates and improved classification performance across 10 benchmarks, with scalability up to 128 layers and 1B parameters.
Standard deep learning relies on Backpropagation (BP), which is constrained by biologically implausible weight symmetry and suffers from significant gradient interference within dense representations. To mitigate these bottlenecks, we propose Selective Adaptive Learning (SAL), a training method that combines selective parameter activation with adaptive area partitioning. Specifically, SAL decomposes the parameter space into mutually exclusive, sample-dependent regions. This decoupling mitigates gradient interference across divergent semantic patterns and addresses explicit weight symmetry requirements through our refined feedback alignment. Empirically, SAL demonstrates competitive convergence rates, leading to improved classification performance across 10 standard benchmarks. Additionally, SAL achieves numerical consistency and competitive accuracy even in deep regimes (up to 128 layers) and large-scale models (up to 1B parameters). Our approach is loosely inspired by biological learning mechanisms, offering a plausible alternative that contributes to the study of scalable neural network training.