Learnability and Competition in High-Dimensional Multi-Component ICA
For researchers in unsupervised learning and ICA, this work provides a theoretical framework for understanding multi-component learning dynamics, though it is incremental as it extends existing mean-field approaches to the multi-component setting.
The paper develops an asymptotically exact mean-field theory for multi-component online ICA, revealing a phase structure with decoupled and competition regimes. It provides explicit learnability boundaries and shows that competition shrinks the stable learning-rate window and increases convergence times, validated on synthetic and hyperspectral data.
Independent Component Analysis (ICA) is a foundational tool for unsupervised representation learning, yet its high-dimensional theory remains largely limited to single-component recovery. We develop an asymptotically exact mean-field theory for multi-component online ICA, capturing the coupling induced by simultaneous learning and orthogonalization. In the high-dimensional limit, the joint empirical distribution of learned estimates and ground-truth components converges to a deterministic process, yielding a closed ODE system for the overlap matrix between learned directions and true components. This characterization reveals a genuinely multi-component, initialization-driven phase structure: a decoupled regime, where estimates align with distinct components and evolve nearly independently, and a competition regime, where overlapping initializations induce orthogonality-driven conflicts, slow reorientation, and delayed convergence. Our steady-state analysis gives explicit learnability boundaries and competition conditions linking step size, data moments, and initialization. These conditions show that larger higher-order moments and competition shrink the stable learning-rate window, increase convergence times, and predict a staircase phenomenon in which the number of recoverable components changes discretely with the learning rate. Experiments on synthetic data and hyperspectral remote sensing data validate the predicted trajectories and phase behavior.