BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
This work addresses the efficiency challenge for LLM deployment by providing empirical insights into method interactions, though it is incremental as it builds on existing quantization and early exit techniques.
The paper tackled the problem of understanding compositional effects of quantization and early exit techniques in Large Language Models, finding that a simple 8-bit quantized model (BitSkip-V1) outperformed more complex variants and matched full-precision baseline quality with a perplexity of 1.13 vs 1.19, while also showing optimal early-exit characteristics with a 32.5% speed gain for minimal quality loss.
The pursuit of efficient Large Language Models (LLMs) has led to increasingly complex techniques like extreme quantization and dynamic routing. While individual benefits of these methods are well-documented, their compositional effects remain poorly understood. This paper introduces BitSkip, a hybrid architectural framework for systematically exploring these interactions. Counter-intuitively, our findings reveal that a simple 8-bit quantized model without Hadamard transform (BitSkip-V1) not only outperforms its more complex 4-bit and Hadamard-enhanced counterparts but also competes the full-precision baseline in quality (perplexity of 1.13 vs 1.19) . The introduction of Hadamard transforms, even at 8-bit precision, catastrophically degraded performance by over 37,000%, tracing fundamental training instability. Our BitSkip-V1 recipe demonstrates superior early-exit characteristics, with layer 18 providing optimal 32.5% speed gain for minimal 4% quality loss.