CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
This work addresses the need for more robust and deployable PEFT methods for adapting large language models under limited compute and memory budgets, representing an incremental advancement over existing low-rank adaptation techniques.
The paper tackled the problem of parameter-efficient fine-tuning (PEFT) for large language models by introducing CTR-LoRA, a framework that integrates rank scheduling with stability-aware optimization, resulting in consistent improvements in accuracy, training stability, reduced memory, and higher throughput on benchmarks for 7B-13B models.
Parameter-efficient fine-tuning (PEFT) has become the standard approach for adapting large language models under limited compute and memory budgets. Although previous methods improve efficiency through low-rank updates, quantization, or heuristic budget reallocation, they often decouple the allocation of capacity from the way updates evolve during training. In this work, we introduce CTR-LoRA, a framework guided by curvature trust region that integrates rank scheduling with stability-aware optimization. CTR-LoRA allocates parameters based on marginal utility derived from lightweight second-order proxies and constrains updates using a Fisher/Hessian-metric trust region. Experiments on multiple open-source backbones (7B-13B), evaluated on both in-distribution and out-of-distribution benchmarks, show consistent improvements over strong PEFT baselines. In addition to increased accuracy, CTR-LoRA enhances training stability, reduces memory requirements, and achieves higher throughput, positioning it on the Pareto frontier of performance and efficiency. These results highlight a principled path toward more robust and deployable PEFT.