Perturbation: A simple and efficient adversarial tracer for representation learning in language models
This addresses a foundational dilemma in representation learning for language models, offering a novel approach to understanding how LMs acquire abstractions.
The authors tackled the problem of finding linguistic representations in language models by introducing a simple adversarial perturbation method that measures how fine-tuning on a single example affects others, revealing structured transfer at multiple linguistic grain sizes without geometric assumptions.
Linguistic representation learning in deep neural language models (LMs) has been studied for decades, for both practical and theoretical reasons. However, finding representations in LMs remains an unsolved problem, in part due to a dilemma between enforcing implausible constraints on representations (e.g., linearity; Arora et al. 2024) and trivializing the notion of representation altogether (Sutter et al., 2025). Here we escape this dilemma by reconceptualizing representations not as patterns of activation but as conduits for learning. Our approach is simple: we perturb an LM by fine-tuning it on a single adversarial example and measure how this perturbation ``infects'' other examples. Perturbation makes no geometric assumptions, and unlike other methods, it does not find representations where it should not (e.g., in untrained LMs). But in trained LMs, perturbation reveals structured transfer at multiple linguistic grain sizes, suggesting that LMs both generalize along representational lines and acquire linguistic abstractions from experience alone.