Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory
This provides a practical path to hardware-efficient kernel memories and insights into robust representation in neural systems, though it appears incremental as it builds on existing KLR methods.
The paper tackles the high computational cost of Kernel Logistic Regression (KLR)-based associative memories by investigating their compressibility through quantization and pruning, finding that the network is extremely robust to low-precision quantization but highly sensitive to pruning.
High-capacity associative memories based on Kernel Logistic Regression (KLR) are known for their exceptional performance but are hindered by high computational costs. This paper investigates the compressibility of KLR-trained Hopfield networks to understand the geometric principles of its robust encoding. We provide a comprehensive geometric theory based on spontaneous symmetry breaking and Walsh analysis, and validate it with compression experiments (quantization and pruning). Our experiments reveal a striking contrast: the network is extremely robust to low-precision quantization but highly sensitive to pruning. Our theory explains this via a ``sparse function, dense representation'' principle, where a sparse input mapping is implemented with a dense, bimodal parameterization. Our findings not only provide a practical path to hardware-efficient kernel memories but also offer new insights into the geometric principles of robust representation in neural systems.