RCStat: A Statistical Framework for using Relative Contextualization in Transformers
This work addresses efficiency and interpretability issues in transformer models for NLP applications, offering incremental improvements over existing methods.
The paper tackles the problem of inefficient attention mechanisms in transformers by introducing RCStat, a statistical framework that uses raw attention logits for relative contextualization, resulting in state-of-the-art key-value compression with minimal quality loss and higher-fidelity attribution across benchmarks.
Prior work on input-token importance in auto-regressive transformers has relied on Softmax-normalized attention weights, which obscure the richer structure of pre-Softmax query-key logits. We introduce RCStat, a statistical framework that harnesses raw attention logits via Relative Contextualization (RC), a random variable measuring contextual alignment between token segments, and derive an efficient upper bound for RC. We demonstrate two applications: (i) Key-Value compression, where RC-based thresholds drive adaptive key-value eviction for substantial cache reduction with minimal quality loss; and (ii) Attribution, where RC yields higher-fidelity token-, sentence-, and chunk-level explanations than post-Softmax methods. Across question answering, summarization, and attribution benchmarks, RCStat achieves significant empirical gains, delivering state-of-the-art compression and attribution performance without any model retraining.