Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation
This addresses the issue of model degradation during deployment for users of optimized language models, offering a lightweight and dataset-agnostic recovery method, though it is incremental as it builds on existing LoRA techniques.
The paper tackles the problem of recovering accuracy in language models degraded by inference optimizations like quantization or serialization, proposing Recover-LoRA, which uses synthetic data and logit distillation to learn LoRA adapters, resulting in accuracy recovery of 5-17% on small language models.
Inference optimizations such as quantization, pruning, format and datatype conversion, model export, and serialization can lead to functional degradations in language model task performance. While most efforts on performance recovery for deployment focus on robust quantization techniques, we focus on recovering model accuracies from any sources that degrade model weights, such as improper model serialization. In this work, we propose Recover-LoRA, a lightweight and dataset agnostic method to recover accuracy in degraded models. Recover-LoRA uses synthetic data and logit distillation to learn LoRA adapters on selective layers that facilitate aligning the degraded model to its full precision model. We investigate the utility of Recover-LoRA across a diverse set of small language models (SLMs), including models with varying attention architectures, multi-head attention (MHA) and group-query attention (GQA), as well as several evaluation datasets. Our results show that Recover-LoRA recovers model accuracies by 5-17% on MHA and GQA SLMs.