Information Subtraction: Learning Representations for Conditional Entropy
This work addresses the need for flexible representation learning to explain relationships between continuous variables, with applications in fairness and domain adaptation, though it builds on prior methods for discrete cases.
The paper tackles the problem of learning representations for conditional entropy and mutual information, particularly for continuous variables, by introducing the Information Subtraction framework, which effectively removes undesired information while preserving desired information, achieving strong performance in fair learning and domain generalization.
The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at https://github.com/jh-liang/Information-Subtraction