EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
This provides an economical and scalable solution for researchers and practitioners to deploy early-exit LLMs, making them more accessible, though it is incremental as it builds on existing early-exit and parameter-efficient tuning methods.
The paper tackles the high computational cost of training early-exit large language models by introducing EE-Tuning, a lightweight method that adds and tunes early-exit layers to pre-trained LLMs with significantly less resources and data, achieving effective inference with limited training budgets.
This work introduces EE-Tuning, a lightweight and economical solution to training/tuning early-exit large language models (LLMs). In contrast to the common approach of full-parameter pre-training, EE-Tuning augments any pre-trained (and possibly fine-tuned) standard LLM with additional early-exit layers that are tuned in a parameter-efficient manner, which requires significantly less computational resources and training data. Our implementation of EE-Tuning achieves outstanding training efficiency via extensive performance optimizations, as well as scalability due to its full compatibility with 3D parallelism. Results of systematic experiments validate the efficacy of EE-Tuning, confirming that effective early-exit LLM inference can be achieved with a limited training budget. In hope of making early-exit LLMs accessible to the community, we release the source code of our implementation of EE-Tuning at https://github.com/pan-x-c/EE-LLM.