Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning
This work addresses the problem of limited domain availability and structural assumption violations in disentangled representation learning for machine learning researchers, offering an incremental improvement by integrating existing assumptions.
The paper tackles the challenge of achieving identifiability in disentangled representation learning by combining sufficient changes in latent variable distributions with sparse mixing procedures, resulting in a less restrictive theory and an estimation framework validated on synthetic and real-world datasets.
Disentangled representation learning aims to uncover latent variables underlying the observed data, and generally speaking, rather strong assumptions are needed to ensure identifiability. Some approaches rely on sufficient changes on the distribution of latent variables indicated by auxiliary variables such as domain indices, but acquiring enough domains is often challenging. Alternative approaches exploit structural sparsity assumptions on the mixing procedure, but such constraints are usually (partially) violated in practice. Interestingly, we find that these two seemingly unrelated assumptions can actually complement each other to achieve identifiability. Specifically, when conditioned on auxiliary variables, the sparse mixing procedure assumption provides structural constraints on the mapping from estimated to true latent variables and hence compensates for potentially insufficient distribution changes. Building on this insight, we propose an identifiability theory with less restrictive constraints regarding distribution changes and the sparse mixing procedure, enhancing applicability to real-world scenarios. Additionally, we develop an estimation framework incorporating a domain encoding network and a sparse mixing constraint and provide two implementations based on variational autoencoders and generative adversarial networks, respectively. Experiment results on synthetic and real-world datasets support our theoretical results.