PiSSA

“Despite its effectiveness and popularity, recent studies have underscored that LoRA and its variants face challenges such as diminishing performance~LoRA, and slower convergence~PiSSA relative to full fine-tuning, which deteriorate further as the rank declines~MoRA,HiRA.”

“SVD-based initialization is computationally expensive and requires a long time”

“SVD-based initialization is only a soft constraint, allowing updates to drift away from the pretrained subspace during training”

“However, it still operates under a fixed low-rank constraint, which restricts its capacity to high-complexity tasks.”

“such SFT-oriented spectral priors can create a fundamental geometric mismatch in RLVR, whose optimization dynamics and effective update patterns differ markedly from SFT”

“they are less pluggable than LoRA, requiring extra computational pipelines and storage for SVD buffers.”

“However, PiSSA operates independently on each layer's weight matrix, missing cross-layer correlations.”

“Recently, PiSSA meng2024pissa proposes to initializing \\( A \\) and \\( B \\) to approximate the original matrix $W$, by performing SVD on $W$. Our method, however, is based on a very different idea, that is to approximate the gradient of $W$, which involves performing SVD on sampled gradients and properly scaling the initialized matrices, as detailed in Section~compare_pissa.”

“OLoRA~buyukakyuz2024olora and PiSSA~meng2024pissa ease optimization by initializing LoRA orthogonally. However, they remove important pre-trained components from the frozen base weights.”

“we enforce the rows of $A$ to be orthonormal by initializing them with right singular vectors of $BA$, which empirically stabilizes training and accelerates optimization compared to a non-orthonormal structure.”

“These approaches primarily operate in weight space and can improve optimization and generalization under moderate ranks. A common assumption underlying these methods is that task-relevant adaptation directions can be inferred directly from the pretrained weight geometry, without explicit reference to data-induced activation patterns.”

“However, these works can only solve either side of the two problems, but do not consider the trade-off between enhancing fine-tuning performance and preserving pre-trained knowledge”