CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion
This work provides a more spectrally efficient parameter-efficient fine-tuning method for large language models, particularly beneficial for practitioners facing computational constraints in complex reasoning tasks.
This paper addresses the "linear ceiling" limitation of Low-Rank Adaptation (LoRA) in complex reasoning tasks, where increasing rank yields diminishing returns. The authors introduce CeRA, a weight-level parallel adapter that uses SiLU gating and structural dropout to expand the manifold, outperforming LoRA at significantly lower ranks (e.g., CeRA rank 64 PPL 3.89 vs. LoRA rank 512 PPL 3.90 on SlimOrca, and CeRA PPL 1.97 vs. LoRA PPL 2.07 on MathInstruct).
Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning (PEFT). However, it faces a critical ``linear ceiling'' in complex reasoning tasks: simply increasing the rank yields diminishing returns due to intrinsic linear constraints. We introduce CeRA (Capacity-enhanced Rank Adaptation), a weight-level parallel adapter that injects SiLU gating and structural dropout to induce manifold expansion. On the SlimOrca benchmark, CeRA breaks this linear barrier: at rank 64 (PPL 3.89), it outperforms LoRA at rank 512 (PPL 3.90), demonstrating superior spectral efficiency. This advantage generalizes to mathematical reasoning, where CeRA achieves a perplexity of 1.97 on MathInstruct, significantly surpassing LoRA's saturation point of 2.07. Mechanism analysis via Singular Value Decomposition (SVD) confirms that CeRA activates the dormant tail of the singular value spectrum, effectively preventing the rank collapse observed in linear methods.