Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters
For practitioners of parameter-efficient fine-tuning, this work provides a controlled comparison showing that finer parameter steps do not guarantee better accuracy-budget curves, highlighting task-dependent behavior.
The paper investigates whether tensorized adapters with finer parameter increments (CP adapters) improve the accuracy-budget trade-off compared to LoRA in low-rank PEFT. On OPT-1.3B, CP adapters fill gaps between LoRA ranks but show task-dependent results: SST-2 plateaus early, BoolQ benefits before saturating below LoRA, and RTE favors LoRA.
Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget. For a $2048{\times}2048$ OPT attention projection, increasing LoRA by one rank stores $4096$ trainable scalars, leaving large gaps between feasible low-budget adapter sizes. This paper asks whether a tensorized adapter with finer capacity increments changes the observed accuracy--budget trade-off. We instantiate this question with fixed-component canonical polyadic (CP) tensor adapters. Under a $32{\times}64{\times}32{\times}64$ tensorization, one normalized CP component stores $193$ trainable scalars per projection, about $21$ times smaller than one LoRA rank step. We compare CP adapters and LoRA on OPT-1.3B across SST-2, RTE, and BoolQ under matched target modules, training protocol, data caps, and seed schedules. CP trains stably and fills the gaps between LoRA ranks, but the effect is task-dependent: SST-2 reaches an early low-budget plateau, BoolQ benefits from additional CP components before saturating slightly below LoRA, and RTE remains LoRA-favored. Finer parameter steps are therefore useful for diagnosing PEFT budget sensitivity, but they do not by themselves guarantee a better accuracy--budget curve.