Adaptive Large Language Models By Layerwise Attention Shortcuts
This addresses the computational inefficiency and inflexibility of stacked transformer layers for large language models, offering a method to improve adaptability across diverse data types.
The paper tackles the rigid sequential processing of transformer architectures by introducing attention shortcuts that allow the final layer to attend to intermediate layers, making models depth- and context-adaptive. It demonstrates superior performance on four datasets (acoustic tokens, natural language, symbolic music) with GPT-like architectures, showing through attention maps that models learn adaptive dependencies across layers.
Transformer architectures are the backbone of the modern AI revolution. However, they are based on simply stacking the same blocks in dozens of layers and processing information sequentially from one block to another. In this paper, we propose to challenge this and introduce adaptive computations for LLM-like setups, which allow the final layer to attend to all of the intermediate layers as it deems fit through the attention mechanism, thereby introducing computational \textbf{attention shortcuts}. These shortcuts can thus make the architecture depth and context adaptive. We showcase four different datasets, namely acoustic tokens, natural language, and symbolic music, and we achieve superior performance for GPT-like architecture. We give evidence via attention maps that the models learn complex dependencies across layers that are adaptive in context and depth depending on the input tokens.