Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
This work addresses the problem of making LLMs more accessible for deployment in resource-constrained environments, representing an incremental advancement by integrating existing techniques.
The paper tackles the computational challenges of fine-tuning and deploying Large Language Models by combining low-rank adapters with Neural Architecture Search, resulting in models with reduced memory footprints and faster inference times.
The rapid expansion of Large Language Models (LLMs) has posed significant challenges regarding the computational resources required for fine-tuning and deployment. Recent advancements in low-rank adapters have demonstrated their efficacy in parameter-efficient fine-tuning (PEFT) of these models. This retrospective paper comprehensively discusses innovative approaches that synergize low-rank representations with Neural Architecture Search (NAS) techniques, particularly weight-sharing super-networks. Robust solutions for compressing and fine-tuning large pre-trained models are developed by integrating these methodologies. Our analysis highlights the potential of these combined strategies to democratize the use of LLMs, making them more accessible for deployment in resource-constrained environments. The resulting models exhibit reduced memory footprints and faster inference times, paving the way for more practical and scalable applications of LLMs. Models and code are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.