Kernel similarity matching with Hebbian neural networks
This work addresses kernel approximation for neural networks, offering a biologically plausible online learning approach, though it is incremental in improving feature learning methods.
The authors tackled the problem of performing kernel similarity matching by directly learning nonlinear features, resulting in a method that generates sparse, selective representations and is comparable or better than random Nyström methods for low-dimensional outputs.
Recent works have derived neural networks with online correlation-based learning rules to perform \textit{kernel similarity matching}. These works applied existing linear similarity matching algorithms to nonlinear features generated with random Fourier methods. In this paper attempt to perform kernel similarity matching by directly learning the nonlinear features. Our algorithm proceeds by deriving and then minimizing an upper bound for the sum of squared errors between output and input kernel similarities. The construction of our upper bound leads to online correlation-based learning rules which can be implemented with a 1 layer recurrent neural network. In addition to generating high-dimensional linearly separable representations, we show that our upper bound naturally yields representations which are sparse and selective for specific input patterns. We compare the approximation quality of our method to neural random Fourier method and variants of the popular but non-biological "Nystr{ö}m" method for approximating the kernel matrix. Our method appears to be comparable or better than randomly sampled Nystr{ö}m methods when the outputs are relatively low dimensional (although still potentially higher dimensional than the inputs) but less faithful when the outputs are very high dimensional.