Adversarial Dependence Minimization
This provides a scalable solution for reducing feature redundancy in machine learning applications, though it builds incrementally on existing decorrelation techniques.
The paper tackles the problem of eliminating nonlinear dependencies between feature dimensions in representation learning, which existing methods fail to address, and demonstrates that their adversarial algorithm improves generalization in image classification and prevents dimensional collapse in self-supervised learning.
Many machine learning techniques rely on minimizing the covariance between output feature dimensions to extract minimally redundant representations from data. However, these methods do not eliminate all dependencies/redundancies, as linearly uncorrelated variables can still exhibit nonlinear relationships. This work provides a differentiable and scalable algorithm for dependence minimization that goes beyond linear pairwise decorrelation. Our method employs an adversarial game where small networks identify dependencies among feature dimensions, while the encoder exploits this information to reduce dependencies. We provide empirical evidence of the algorithm's convergence and demonstrate its utility in three applications: extending PCA to nonlinear decorrelation, improving the generalization of image classification methods, and preventing dimensional collapse in self-supervised representation learning.