Dependence Induced Representations
This work addresses foundational representation learning challenges for machine learning researchers, offering theoretical insights into dependence-based features and their practical implications.
The paper tackles the problem of learning feature representations from dependent random variables, establishing conditions for dependence-induced representations and linking them to maximal correlation functions and minimal sufficient statistics. It shows that a broad family of loss functions can learn such representations, revealing connections between different losses and providing a statistical interpretation of neural collapse in deep classifiers.
We study the problem of learning feature representations from a pair of random variables, where we focus on the representations that are induced by their dependence. We provide sufficient and necessary conditions for such dependence induced representations, and illustrate their connections to Hirschfeld--Gebelein--Rényi (HGR) maximal correlation functions and minimal sufficient statistics. We characterize a large family of loss functions that can learn dependence induced representations, including cross entropy, hinge loss, and their regularized variants. In particular, we show that the features learned from this family can be expressed as the composition of a loss-dependent function and the maximal correlation function, which reveals a key connection between representations learned from different losses. Our development also gives a statistical interpretation of the neural collapse phenomenon observed in deep classifiers. Finally, we present the learning design based on the feature separation, which allows hyperparameter tuning during inference.