SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
This work addresses the challenge of scalable and efficient adaptation of LLMs for diverse tasks, offering a novel approach that is incremental but impactful for reducing resource usage in AI deployment.
The paper tackles the problem of efficiently adapting large language models to new contexts by proposing SHINE, a scalable hypernetwork that maps contexts into LoRA adapters in a single forward pass, achieving strong performance while saving time, computation, and memory costs compared to fine-tuning-based methods.
We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLM). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/Yewei-Liu/SHINE