Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
For researchers using cross-attention to study multilingual NMT, this artifact makes uncorrected analyses unreliable, and the paper provides a validated mitigation.
The paper discovers that in NLLB-200, non-content tokens capture 83-91% of cross-attention mass, causing raw metrics to underestimate content-level similarity by nearly half (36.7% vs 70.7%). A content-only filtering method recovers linguistic signals, revealing a 16.9 percentage-point gap between teacher-forcing and generation modes.
Cross-attention patterns in neural machine translation (NMT) are widely used to study how multilingual models align linguistic structure. We report a systematic artifact in cross-attention analysis of NLLB-200 (600M): non-content tokens - primarily end-of-sequence tokens, language tags, and punctuation - capture 83 percent to 91 percent of total cross-attention mass. We term these "attention sinks," extending findings from LLMs [Xiao et al., 2023] to NMT cross-attention and identifying a causal mechanism rooted in vocabulary design rather than position bias. This artifact causes raw metrics to underestimate content-level similarity by nearly half (36.7 percent raw vs. 70.7 percent filtered), rendering uncorrected analyses unreliable. To address this, we validate a content-only filtering methodology that removes non-content tokens and renormalizes the distribution. Applying this to 1,000 parallel sentences across African languages (Swahili, Kikuyu, Somali, Luo) and non-African benchmarks (German, Turkish, Chinese, Hindi), we confirm the artifact is universal and recover masked linguistic signals: a 16.9 percentage-point gap between teacher-forcing and generation modes, clear language-family clustering in attention entropy, and a hidden Somali paradox linking SOV word order to monotonic alignment. We release our filtering toolkit and corrected datasets to support reproducible interpretability research on multilingual NMT.