Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors
This work addresses the challenge of model selection for ill-defined qualities like disentanglement in probabilistic generative models, which is incremental as it builds on existing theoretical characterizations.
The paper tackles the problem of evaluating disentanglement in generative models without requiring labeled latent factors, by introducing a method based on training dynamics that correlates with supervised metrics and serves as an unsupervised indicator for downstream tasks like reinforcement learning and fairness classification.
Probabilistic generative models provide a flexible and systematic framework for learning the underlying geometry of data. However, model selection in this setting is challenging, particularly when selecting for ill-defined qualities such as disentanglement or interpretability. In this work, we address this gap by introducing a method for ranking generative models based on the training dynamics exhibited during learning. Inspired by recent theoretical characterizations of disentanglement, our method does not require supervision of the underlying latent factors. We evaluate our approach by demonstrating the need for disentanglement metrics which do not require labels\textemdash the underlying generative factors. We additionally demonstrate that our approach correlates with baseline supervised methods for evaluating disentanglement. Finally, we show that our method can be used as an unsupervised indicator for downstream performance on reinforcement learning and fairness-classification problems.