Analyzing Generative Models by Manifold Entropic Metrics
This work addresses the challenge of evaluating generative models for interpretability, which is important for researchers and practitioners in machine learning, though it is incremental as it builds on existing principles like independent mechanisms.
The paper tackled the problem of objectively measuring desirable properties like disentanglement in generative models by introducing a novel set of information-theoretic evaluation metrics, and demonstrated their usefulness through experiments on EMNIST, ranking model architectures and training procedures based on their inductive bias for aligned and disentangled representations.
Good generative models should not only synthesize high quality data, but also utilize interpretable representations that aid human understanding of their behavior. However, it is difficult to measure objectively if and to what degree desirable properties of disentangled representations have been achieved. Inspired by the principle of independent mechanisms, we address this difficulty by introducing a novel set of tractable information-theoretic evaluation metrics. We demonstrate the usefulness of our metrics on illustrative toy examples and conduct an in-depth comparison of various normalizing flow architectures and $β$-VAEs on the EMNIST dataset. Our method allows to sort latent features by importance and assess the amount of residual correlations of the resulting concepts. The most interesting finding of our experiments is a ranking of model architectures and training procedures in terms of their inductive bias to converge to aligned and disentangled representations during training.