Quantitative Attractor Analysis of High-Capacity Kernel Logistic Regression Hopfield Networks
This provides empirical design principles for high-capacity associative memories, addressing a domain-specific need in neural network modeling.
The paper tackled the problem of understanding the performance and stability of kernel-based Hopfield networks, revealing that Kernel Logistic Regression and Kernel Ridge Regression achieve high storage capacities with linear scaling relative to network size under optimized kernel width scaling.
Kernel-based learning methods such as Kernel Logistic Regression (KLR) can dramatically increase the storage capacity of Hopfield networks, but the principles governing their performance and stability remain largely uncharacterized. This paper presents a comprehensive quantitative analysis of the attractor landscape in KLR-trained networks to establish a solid foundation for their design and application. Through extensive, statistically validated simulations, we address critical questions of generality, scalability, and robustness. Our comparative analysis reveals that KLR and Kernel Ridge Regression (KRR) exhibit similarly high storage capacities and clean attractor landscapes, suggesting this is a general property of kernel regression methods, though KRR is computationally much faster. We uncover a non-trivial, scale-dependent scaling law for the kernel width ($γ$), demonstrating that optimal capacity requires gamma to be scaled such that $γ\times N$ increases with network size $N$. This implies that larger networks necessitate more localized kernels -- where each pattern's influence is more spatially confined--to manage inter-pattern interference. Under this optimized scaling, we provide definitive evidence that the storage capacity scales linearly with network size ($P \propto N$). Furthermore, our sensitivity analysis shows that performance is remarkably robust to the choice of the regularization parameter lambda. Collectively, these findings provide a clear set of empirical principles for designing high-capacity, robust associative memories and clarify the mechanisms that enable kernel methods to overcome the classical limitations of Hopfield-type models.