Normalized Narrow Jump To Conclusions: Normalized Narrow Shortcuts for Parameter Efficient Early Exit Transformer Prediction
This work addresses the problem of high inference costs for large language models by enabling more parameter-efficient early exit strategies, representing an incremental improvement in model efficiency.
The paper tackles the computational expense of linear shortcutting for early exit in large transformer models by proposing Narrow Jump to Conclusions (NJTC) and Normalized Narrow Jump to Conclusions (N-NJTC), which reduce shortcut parameter count by over 97% while reliably outperforming identity shortcuts and offering stable precision across transformer block levels for models like GPT-2-XL, Phi3-Mini, and Llama2-7B.
With the size and cost of large transformer-based language models growing, recently, there has been interest in shortcut casting of early transformer hidden-representations to final-representations for cheaper model inference. In particular, shortcutting pre-trained transformers with linear transformations over early layers has been shown to improve precision in early inference. However, for large language models, even this becomes computationally expensive. In this work, we propose Narrow Jump to Conclusions (NJTC) and Normalized Narrow Jump to Conclusions (N-NJTC) - parameter efficient alternatives to standard linear shortcutting that reduces shortcut parameter count by over 97%. We show that N-NJTC reliably outperforms Identity shortcuts at early stages and offers stable precision from all transformer block levels for GPT-2-XL, Phi3-Mini and Llama2-7B transformer models, demonstrating the viability of more parameter efficient short-cutting approaches.