AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
For practitioners fine-tuning large language models with LoRA, AdaPaD eliminates the need to pre-specify rank and reduces adapter size without sacrificing performance.
AdaPaD introduces a parallel deflation method for LoRA fine-tuning that trains all rank-1 components simultaneously with self-correcting errors, achieving competitive performance on GLUE with DeBERTaV3-base and on Qwen3-0.6B SQuAD/SQuAD v2 while using adapters that are 30.7% smaller on average.
Fine-tuning large language models with LoRA requires choosing a rank r before training starts. Existing approaches either extract rank-1 components sequentially, freezing each component's error permanently into every subsequent residual, or optimize the full low-rank factorization jointly with guarantees that describe only the joint update, not individual rank-1 directions. We present AdaPaD (Adaptive Parallel Deflation), which trains all rank-1 components simultaneously: each worker refines its component against a deflation target built from the latest estimates of all predecessors, and as those estimates improve, the targets improve too. We call this property self-correction: deflation errors converge to zero over rounds rather than persisting as fixed residuals. On top of this backbone, AdaPaD adds advance learning (private pre-training before activation) and per-module dynamic rank discovery (importance-based growth until a shared budget is exhausted), making the rank distribution an output rather than an input. We prove that every component's error decays exponentially after a warm-up period, with a generalization bound that splits into a vanishing algorithmic term and an irreducible statistical floor. Empirically, AdaPaD is competitive with adaptive-rank LoRA baselines on GLUE with DeBERTaV3-base at matched parameter budgets, and competitive with fixed-rank LoRA on Qwen3-0.6B SQuAD/SQuAD v2 while deploying an adapter that is on average 30.7% smaller.