On the Versatile Uses of Partial Distance Correlation in Deep Learning
This work addresses the difficulty in comparing neural network functions for researchers and practitioners, offering a method that can enhance model understanding and optimization, though it appears incremental as it adapts an existing statistical concept to deep learning.
The paper tackles the problem of systematically comparing the functional behavior of neural network models by introducing partial distance correlation as a versatile tool for evaluating correlations between feature spaces of different dimensions, enabling applications such as conditioning models, learning disentangled representations, and improving adversarial robustness.
Comparing the functional behavior of neural network models, whether it is a single network over time or two (or more networks) during or post-training, is an essential step in understanding what they are learning (and what they are not), and for identifying strategies for regularization or efficiency improvements. Despite recent progress, e.g., comparing vision transformers to CNNs, systematic comparison of function, especially across different networks, remains difficult and is often carried out layer by layer. Approaches such as canonical correlation analysis (CCA) are applicable in principle, but have been sparingly used so far. In this paper, we revisit a (less widely known) from statistics, called distance correlation (and its partial variant), designed to evaluate correlation between feature spaces of different dimensions. We describe the steps necessary to carry out its deployment for large scale models -- this opens the door to a surprising array of applications ranging from conditioning one deep model w.r.t. another, learning disentangled representations as well as optimizing diverse models that would directly be more robust to adversarial attacks. Our experiments suggest a versatile regularizer (or constraint) with many advantages, which avoids some of the common difficulties one faces in such analyses. Code is at https://github.com/zhenxingjian/Partial_Distance_Correlation.