Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning
This provides a new theoretical framework for designing identifiable representation learning methods by exploiting inductive biases on mechanisms, potentially broadening the scope beyond existing assumptions like independence.
The paper tackles the problem of identifying latent properties in unsupervised representation learning by leveraging knowledge of the mechanisms governing their evolution, proving that exact knowledge allows identification up to shared equivariances and generalizing to settings with hypothesis classes or stochastic mechanisms.
A key goal of unsupervised representation learning is "inverting" a data generating process to recover its latent properties. Existing work that provably achieves this goal relies on strong assumptions on relationships between the latent variables (e.g., independence conditional on auxiliary information). In this paper, we take a very different perspective on the problem and ask, "Can we instead identify latent properties by leveraging knowledge of the mechanisms that govern their evolution?" We provide a complete characterization of the sources of non-identifiability as we vary knowledge about a set of possible mechanisms. In particular, we prove that if we know the exact mechanisms under which the latent properties evolve, then identification can be achieved up to any equivariances that are shared by the underlying mechanisms. We generalize this characterization to settings where we only know some hypothesis class over possible mechanisms, as well as settings where the mechanisms are stochastic. We demonstrate the power of this mechanism-based perspective by showing that we can leverage our results to generalize existing identifiable representation learning results. These results suggest that by exploiting inductive biases on mechanisms, it is possible to design a range of new identifiable representation learning approaches.