How transformers learn structured data: insights from hierarchical filtering
This provides insights into the interpretability of transformers for structured data, though it appears incremental as it builds on existing understanding of transformer mechanisms.
The researchers investigated how transformers learn structured data by creating a hierarchical filtering procedure to control positional correlations in sequences on trees, finding that vanilla encoder-only transformers can approximate exact inference algorithms and that correlations at larger distances are sequentially included during training.
Understanding the learning process and the embedded computation in transformers is becoming a central goal for the development of interpretable AI. In the present study, we introduce a hierarchical filtering procedure for data models of sequences on trees, allowing us to hand-tune the range of positional correlations in the data. Leveraging this controlled setting, we provide evidence that vanilla encoder-only transformers can approximate the exact inference algorithm when trained on root classification and masked language modeling tasks, and study how this computation is discovered and implemented. We find that correlations at larger distances, corresponding to increasing layers of the hierarchy, are sequentially included by the network during training. By comparing attention maps from models trained with varying degrees of filtering and by probing the different encoder levels, we find clear evidence of a reconstruction of correlations on successive length scales corresponding to the various levels of the hierarchy, which we relate to a plausible implementation of the exact inference algorithm within the same architecture.