Segment-Based Attention Masking for GPTs
This addresses a bottleneck in language model inference efficiency for users of GPT-based systems, though it is an incremental improvement over existing methods.
The paper tackles the inefficiency of causal attention masking during the prefill phase in GPT models by introducing segment-based masking that treats prompt blocks as units, allowing non-causal access within segments without extra computational cost. It achieves state-of-the-art performance when integrated into models like Llama and Qwen.
Modern Language Models (LMs) owe much of their success to masked causal attention, the backbone of Generative Pre-Trained Transformer (GPT) models. Although GPTs can process the entire user prompt at once, the causal masking is applied to all input tokens step-by-step, mimicking the generation process. This imposes an unnecessary constraint during the initial "prefill" phase when the model processes the input prompt and generates the internal representations before producing any output tokens. In this work, attention is masked based on the known block structure at the prefill phase, followed by the conventional token-by-token autoregressive process after that. For example, in a typical chat prompt, the system prompt is treated as one block, and the user prompt as the next one. Each of these is treated as a unit for the purpose of masking, such that the first tokens in each block can access the subsequent tokens in a non-causal manner. Then, the model answer is generated in the conventional causal manner. This Segment-by-Segment scheme entails no additional computational overhead. When integrating it into models such as Llama and Qwen, state-of-the-art performance is consistently achieved.