Measuring Orthogonality in Representations of Generative Models
This work addresses the need for better evaluation metrics in unsupervised learning, offering a more flexible alternative to disentanglement metrics that can overlook high-quality representations, potentially improving model assessment and development.
The paper tackles the problem of evaluating unsupervised representation learning by proposing two new metrics, Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR), which measure orthogonality and rank of generative factor subspaces; these metrics consistently show stronger correlations with downstream task performance than traditional disentanglement metrics across benchmark datasets and models.
In unsupervised representation learning, models aim to distill essential features from high-dimensional data into lower-dimensional learned representations, guided by inductive biases. Understanding the characteristics that make a good representation remains a topic of ongoing research. Disentanglement of independent generative processes has long been credited with producing high-quality representations. However, focusing solely on representations that adhere to the stringent requirements of most disentanglement metrics, may result in overlooking many high-quality representations, well suited for various downstream tasks. These metrics often demand that generative factors be encoded in distinct, single dimensions aligned with the canonical basis of the representation space. Motivated by these observations, we propose two novel metrics: Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR). These metrics evaluate the mutual orthogonality and rank of generative factor subspaces. Throughout extensive experiments on common downstream tasks, over several benchmark datasets and models, IWO and IWR consistently show stronger correlations with downstream task performance than traditional disentanglement metrics. Our findings suggest that representation quality is closer related to the orthogonality of independent generative processes rather than their disentanglement, offering a new direction for evaluating and improving unsupervised learning models.