Stochastic algorithms with descent guarantees for ICA
This work addresses the challenge of hyper-parameter sensitivity and lack of convergence guarantees in ICA algorithms, which is important for researchers and practitioners in signal processing and machine learning dealing with large-scale data, though it is incremental as it builds on existing majorization-minimization techniques.
The paper tackles the non-convex optimization problem in Independent Component Analysis (ICA) by developing a new majorization-minimization framework for the Infomax loss function, resulting in online and incremental algorithms that eliminate the need for hyper-parameter tuning and guarantee loss decrease, with experiments showing state-of-the-art improvements on large-scale datasets.
Independent component analysis (ICA) is a widespread data exploration technique, where observed signals are modeled as linear mixtures of independent components. From a machine learning point of view, it amounts to a matrix factorization problem with a statistical independence criterion. Infomax is one of the most used ICA algorithms. It is based on a loss function which is a non-convex log-likelihood. We develop a new majorization-minimization framework adapted to this loss function. We derive an online algorithm for the streaming setting, and an incremental algorithm for the finite sum setting, with the following benefits. First, unlike most algorithms found in the literature, the proposed methods do not rely on any critical hyper-parameter like a step size, nor do they require a line-search technique. Second, the algorithm for the finite sum setting, although stochastic, guarantees a decrease of the loss function at each iteration. Experiments demonstrate progress on the state-of-the-art for large scale datasets, without the necessity for any manual parameter tuning.