A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
This work addresses fair classification by improving disentanglement of sensitive attributes, offering a parameter-free method that is easier to train, though it is incremental as it builds on existing disentanglement techniques.
The authors tackled the problem of learning disentangled representations for fair classification by proposing a novel information-theoretic regularizer called CLINIC, which minimizes mutual information between latent representations and sensitive attributes conditional on the target, resulting in a better disentanglement/accuracy trade-off and improved generalization compared to previous methods, as demonstrated through training over 2,000 neural networks.
One of the pursued objectives of deep learning is to provide tools that learn abstract representations of reality from the observation of multiple contextual situations. More precisely, one wishes to extract disentangled representations which are (i) low dimensional and (ii) whose components are independent and correspond to concepts capturing the essence of the objects under consideration (Locatello et al., 2019b). One step towards this ambitious project consists in learning disentangled representations with respect to a predefined (sensitive) attribute, e.g., the gender or age of the writer. Perhaps one of the main application for such disentangled representations is fair classification. Existing methods extract the last layer of a neural network trained with a loss that is composed of a cross-entropy objective and a disentanglement regularizer. In this work, we adopt an information-theoretic view of this problem which motivates a novel family of regularizers that minimizes the mutual information between the latent representation and the sensitive attribute conditional to the target. The resulting set of losses, called CLINIC, is parameter free and thus, it is easier and faster to train. CLINIC losses are studied through extensive numerical experiments by training over 2k neural networks. We demonstrate that our methods offer a better disentanglement/accuracy trade-off than previous techniques, and generalize better than training with cross-entropy loss solely provided that the disentanglement task is not too constraining.