Higher-Order Modular Attention: Fusing Pairwise and Triadic Interactions for Protein Sequences
This work addresses a domain-specific problem for protein sequence prediction by providing an incremental improvement through explicit triadic terms.
The paper tackled the problem of capturing cooperative dependencies among three or more residues in protein sequences, which standard pairwise attention in Transformers does not explicitly handle, by introducing Higher-Order Modular Attention (HOMA) that fuses pairwise and triadic interactions, resulting in consistent improvements across three TAPE benchmarks for Secondary Structure, Fluorescence, and Stability compared to standard and efficient attention variants.
Transformer self-attention computes pairwise token interactions, yet protein sequence to phenotype relationships often involve cooperative dependencies among three or more residues that dot product attention does not capture explicitly. We introduce Higher-Order Modular Attention, HOMA, a unified attention operator that fuses pairwise attention with an explicit triadic interaction pathway. To make triadic attention practical on long sequences, HOMA employs block-structured, windowed triadic attention. We evaluate on three TAPE benchmarks for Secondary Structure, Fluorescence, and Stability. Our attention mechanism yields consistent improvements across all tasks compared with standard self-attention and efficient variants including block-wise attention and Linformer. These results suggest that explicit triadic terms provide complementary representational capacity for protein sequence prediction at controllable additional computational cost.