Can Interpretation Predict Behavior on Unseen Data?
This provides a proof-of-concept for using interpretability to predict unseen model behavior, which could aid in improving model reliability, though it is incremental as it focuses on synthetic settings.
The paper investigates whether interpretability tools can predict how Transformer models generalize to out-of-distribution (OOD) data, finding that hierarchical attention patterns in-distribution correlate with hierarchical generalization OOD in synthetic classification tasks.
Interpretability research often aims to predict how a model will respond to targeted interventions on specific mechanisms. However, it rarely predicts how a model will respond to unseen input data. This paper explores the promises and challenges of interpretability as a tool for predicting out-of-distribution (OOD) model behavior. Specifically, we investigate the correspondence between attention patterns and OOD generalization in hundreds of Transformer models independently trained on a synthetic classification task. These models exhibit several distinct systematic generalization rules OOD, forming a diverse population for correlational analysis. In this setting, we find that simple observational tools from interpretability can predict OOD performance. In particular, when in-distribution attention exhibits hierarchical patterns, the model is likely to generalize hierarchically on OOD data -- even when the rule's implementation does not rely on these hierarchical patterns, according to ablation tests. Our findings offer a proof-of-concept to motivate further interpretability work on predicting unseen model behavior.