When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
This work addresses a foundational problem in understanding neural network representations for researchers in machine learning, though it is incremental as it builds on identifiability theory to refine existing similarity measures.
The paper investigates the conditions under which deep neural networks with similar output distributions have similar internal representations, proving that small KL divergence does not guarantee representational similarity and defining a new distance metric that does. Experiments on CIFAR-10 and synthetic data show that models with near-maximum likelihood can learn dissimilar representations, while wider networks exhibit closer distributions and more similar representations under the proposed metric.
When and why representations learned by different deep neural networks are similar is an active research topic. We choose to address these questions from the perspective of identifiability theory, which suggests that a measure of representational similarity should be invariant to transformations that leave the model distribution unchanged. Focusing on a model family which includes several popular pre-training approaches, e.g., autoregressive language models, we explore when models which generate distributions that are close have similar representations. We prove that a small Kullback--Leibler divergence between the model distributions does not guarantee that the corresponding representations are similar. This has the important corollary that models with near-maximum data likelihood can still learn dissimilar representations -- a phenomenon mirrored in our experiments with models trained on CIFAR-10. We then define a distributional distance for which closeness implies representational similarity, and in synthetic experiments, we find that wider networks learn distributions which are closer with respect to our distance and have more similar representations. Our results thus clarify the link between closeness in distribution and representational similarity.