Independent Mechanism Analysis and the Manifold Hypothesis
This work addresses a theoretical problem in representation learning for researchers, but it is incremental as it builds on prior IMA work.
The paper tackles non-identifiability in nonlinear Independent Component Analysis by extending Independent Mechanism Analysis to settings with more observed mixtures than latent components, aligned with the manifold hypothesis, and shows it circumvents non-identifiability issues while proving it is approximately satisfied with high probability under random conditions.
Independent Mechanism Analysis (IMA) seeks to address non-identifiability in nonlinear Independent Component Analysis (ICA) by assuming that the Jacobian of the mixing function has orthogonal columns. As typical in ICA, previous work focused on the case with an equal number of latent components and observed mixtures. Here, we extend IMA to settings with a larger number of mixtures that reside on a manifold embedded in a higher-dimensional than the latent space -- in line with the manifold hypothesis in representation learning. For this setting, we show that IMA still circumvents several non-identifiability issues, suggesting that it can also be a beneficial principle for higher-dimensional observations when the manifold hypothesis holds. Further, we prove that the IMA principle is approximately satisfied with high probability (increasing with the number of observed mixtures) when the directions along which the latent components influence the observations are chosen independently at random. This provides a new and rigorous statistical interpretation of IMA.