Intervening to learn and compose disentangled representations
This work addresses a foundational problem in machine learning by potentially improving generative models for tasks requiring structured and interpretable representations, though it appears incremental as it builds on existing encoder-decoder frameworks.
The paper tackles the tension between expressivity and structure in generative models by proposing a new approach that adds a decoder-only module to learn disentangled latent representations, enabling out-of-distribution generation and proving an identifiability result for nonlinear models.
In designing generative models, it is commonly believed that in order to learn useful latent structure, we face a fundamental tension between expressivity and structure. In this paper we challenge this view by proposing a new approach to training arbitrarily expressive generative models that simultaneously learn disentangled latent structure. This is accomplished by adding a simple decoder-only module to the head of an existing decoder block that can be arbitrarily complex. The module learns to process concept information by implicitly inverting linear representations from an encoder. Inspired by the notion of intervention in causal graphical models, our module selectively modifies its architecture during training, allowing it to learn a compact joint model over different contexts. We show how adding this module leads to disentangled representations that can be composed for out-of-distribution generation. To further validate our proposed approach, we prove a new identifiability result that extends existing work on identifying structured representations in nonlinear models.