Attend or Perish: Benchmarking Attention in Algorithmic Reasoning
This work addresses the reliability of algorithmic reasoning in transformers for researchers and practitioners, providing a tool to distinguish understanding from memorization, though it is incremental in benchmarking and interpretability methods.
The paper tackles the problem of assessing whether transformers genuinely understand algorithmic tasks or merely memorize patterns, by introducing AttentionSpan, a benchmark with infinite input domains to test extrapolation and robustness. The result shows that attention mechanisms directly cause failures in extrapolation, as demonstrated through analysis of attention maps and interventions.
Can transformers learn to perform algorithmic tasks reliably across previously unseen input/output domains? While pre-trained language models show solid accuracy on benchmarks incorporating algorithmic reasoning, assessing the reliability of these results necessitates an ability to distinguish genuine algorithmic understanding from memorization. In this paper, we propose AttentionSpan, an algorithmic benchmark comprising five tasks of infinite input domains where we can disentangle and trace the correct, robust algorithm necessary for the task. This allows us to assess (i) models' ability to extrapolate to unseen types of inputs, including new lengths, value ranges or input domains, but also (ii)to assess the robustness of their learned mechanisms. By analyzing attention maps and performing targeted interventions, we show that attention mechanism directly causes failures in extrapolation. We make the implementation of all our tasks and interpretability methods publicly available at https://github.com/michalspiegel/AttentionSpan .