Self-calibrating Neural Networks for Dimensionality Reduction
This work addresses a specific bottleneck in streaming data analysis for researchers in machine learning and neuroscience, representing an incremental improvement over prior methods.
The paper tackles the problem of adaptively determining the number of output dimensions in online dimensionality reduction algorithms by introducing self-calibrating thresholds based on singular values, resulting in effective performance demonstrated mathematically and in simulations.
Recently, a novel family of biologically plausible online algorithms for reducing the dimensionality of streaming data has been derived from the similarity matching principle. In these algorithms, the number of output dimensions can be determined adaptively by thresholding the singular values of the input data matrix. However, setting such threshold requires knowing the magnitude of the desired singular values in advance. Here we propose online algorithms where the threshold is self-calibrating based on the singular values computed from the existing observations. To derive these algorithms from the similarity matching cost function we propose novel regularizers. As before, these online algorithms can be implemented by Hebbian/anti-Hebbian neural networks in which the learning rule depends on the chosen regularizer. We demonstrate both mathematically and via simulation the effectiveness of these online algorithms in various settings.