Rethinking Disentanglement under Dependent Factors of Variation
This addresses a foundational limitation in representation learning for AI/ML by enabling more realistic disentanglement evaluation, though it is incremental as it builds on prior definitions.
The paper tackles the problem that existing disentanglement definitions and metrics assume independent factors of variation, which is unrealistic in real-world scenarios, by proposing a new information-theoretic definition and measurement method that works with dependent factors, showing it correctly measures disentanglement while other methods fail.
Representation learning is an approach that allows to discover and extract the factors of variation from the data. Intuitively, a representation is said to be disentangled if it separates the different factors of variation in a way that is understandable to humans. Definitions of disentanglement and metrics to measure it usually assume that the factors of variation are independent of each other. However, this is generally false in the real world, which limits the use of these definitions and metrics to very specific and unrealistic scenarios. In this paper we give a definition of disentanglement based on information theory that is also valid when the factors of variation are not independent. Furthermore, we relate this definition to the Information Bottleneck Method. Finally, we propose a method to measure the degree of disentanglement from the given definition that works when the factors of variation are not independent. We show through different experiments that the method proposed in this paper correctly measures disentanglement with non-independent factors of variation, while other methods fail in this scenario.