Attention-likelihood relationship in transformers
This work addresses the robustness of LLMs for real-world applications, but it is incremental as it builds on existing transformer analysis.
The study investigated how large language models represent out-of-context words by analyzing the correlation between token likelihood and attention values in transformers, finding that unexpected tokens lead to reduced self-attention in higher layers.
We analyze how large language models (LLMs) represent out-of-context words, investigating their reliance on the given context to capture their semantics. Our likelihood-guided text perturbations reveal a correlation between token likelihood and attention values in transformer-based language models. Extensive experiments reveal that unexpected tokens cause the model to attend less to the information coming from themselves to compute their representations, particularly at higher layers. These findings have valuable implications for assessing the robustness of LLMs in real-world scenarios. Fully reproducible codebase at https://github.com/Flegyas/AttentionLikelihood.