AdaLoRA

“AdaLoRA uses a single dataset to simultaneously learn rank-1 matrices and their importance scores, which can easily lead to overfitting.”

“leaves pretraining and finetuning completely intact. There is no need for additional objectives or regularizers, which can slow down convergence and affect the training optimum reached.”

“However, the scoring mechanism in AdaLoRA is primarily based on instantaneous gradient signals, which fail to capture long-term parameter contributions and inter-layer interactions.”

“Methods such as AdaLoRA Zhang2023AdaptiveBA rely on sensitivity-based importance scores that are unreliable as they only consider how a single parameter change affects the model under the assumption that no other parameters change.”

“LoRA and AdaLoRA still clearly overfit the training data as fine-tuning advances, with decreases in training losses but increases in test losses.”

“However, it is worth noting that AdaLoRA entails additional computational overhead and necessitates a higher initial budget of trainable parameters, making it less suitable for low-resource scenarios.”

“AdaLoRA zhang2023adalora redistributes per-layer rank dynamically but recomputes SVD-based importance at every step and introduces multiple schedule hyperparameters.”

“As fine-tuning progresses, the disparity between training and testing losses in both LoRA and AdaLoRA becomes more pronounced. Beyond a certain number of iterations, we observe an increase in test losses alongside a continued decrease in training losses, clearly indicating a tendency for LoRA and AdaLoRA to overfit to the training data.”

“they do not couple rank allocation to changes in dense computation, and therefore do not directly optimize for inference efficiency.”

“Although these approaches reduce redundancy and improve efficiency, they remain constrained by fixed ranks and show limited flexibility and generalization across datasets and architectures.”

“introduces orthogonal regularization to ensure that the low-rank projection matrices comply with Singular Value Decomposition (SVD), thus avoiding the reliance on incremental updates, albeit at the cost of increased computational complexity.”

“This is a form of structured pruning that tackles the symptom of wasted capacity caused by rank collapse. In contrast, our approach aims to prevent rank collapse by construction.”