Internal Representations of Vision Models Through the Lens of Frames on Data Manifolds
This work offers a novel analytical tool for researchers studying representation geometry in vision models, though it is incremental in advancing existing understanding.
The authors tackled the problem of understanding how model design choices affect internal representations in deep learning by introducing neural frames, a method inspired by frames on tangent bundles, to analyze perturbations like augmentations and noise. They provided new insights into phenomena such as invariance from augmentation and trade-offs in adversarial training.
While the last five years have seen considerable progress in understanding the internal representations of deep learning models, many questions remain. This is especially true when trying to understand the impact of model design choices, such as model architecture or training algorithm, on hidden representation geometry and dynamics. In this work we present a new approach to studying such representations inspired by the idea of a frame on the tangent bundle of a manifold. Our construction, which we call a neural frame, is formed by assembling a set of vectors representing specific types of perturbations of a data point, for example infinitesimal augmentations, noise perturbations, or perturbations produced by a generative model, and studying how these change as they pass through a network. Using neural frames, we make observations about the way that models process, layer-by-layer, specific modes of variation within a small neighborhood of a datapoint. Our results provide new perspectives on a number of phenomena, such as the manner in which training with augmentation produces model invariance or the proposed trade-off between adversarial training and model generalization.