Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions
This work addresses the fundamental problem of understanding feature learning mechanisms in deep neural networks for researchers in machine learning theory, though it is incremental as it builds on known connections between ICA and neural networks.
The paper investigates feature learning from non-Gaussian inputs by comparing Independent Component Analysis (ICA) and stochastic gradient descent (SGD), showing that FastICA requires at least n ≳ d^4 samples to recover a non-Gaussian direction, while SGD can achieve optimal sample complexity of n ≳ d^2 with smoothed loss.
Deep neural networks learn structured features from complex, non-Gaussian inputs, but the mechanisms behind this process remain poorly understood. Our work is motivated by the observation that the first-layer filters learnt by deep convolutional neural networks from natural images resemble those learnt by independent component analysis (ICA), a simple unsupervised method that seeks the most non-Gaussian projections of its inputs. This similarity suggests that ICA provides a simple, yet principled model for studying feature learning. Here, we leverage this connection to investigate the interplay between data structure and optimisation in feature learning for the most popular ICA algorithm, FastICA, and stochastic gradient descent (SGD), which is used to train deep networks. We rigorously establish that FastICA requires at least $n\gtrsim d^4$ samples to recover a single non-Gaussian direction from $d$-dimensional inputs on a simple synthetic data model. We show that vanilla online SGD outperforms FastICA, and prove that the optimal sample complexity $n \gtrsim d^2$ can be reached by smoothing the loss, albeit in a data-dependent way. We finally demonstrate the existence of a search phase for FastICA on ImageNet, and discuss how the strong non-Gaussianity of said images compensates for the poor sample complexity of FastICA.