DoRA

“DoRA decomposes the model weights into their directional and magnitude components and fine-tunes both, but only the former remains low-rank.”

“This incurs (d_in^2) memory for the identity matrix alone: 32 MB at d_in=4096, 128 MB at d_in=8192 in bf16. Including the dense BA product and composed-weight copy, a single module allocates 3–4 dense [d_out, d_in] temporaries: 512 MB at d_in=8192.”

“DoRA implicitly assumes that the direction of a matrix can be decomposed into per-column units, an assumption that lacks a clear theoretical grounding in matrix analysis.”

“DoRA relies on strict normalization, which makes it sensitive to optimization instabilities: when the adapted weight norm approaches zero, gradients can explode, destabilizing training.”

“Nonetheless, DoRA introduces additional parameters and over-expressive architecture compared to LoRA, which can exacerbate overfitting issues when adapting to small downstream datasets (See tab:gap).”

“These approaches primarily operate in weight space and can improve optimization and generalization under moderate ranks. A common assumption underlying these methods is that task-relevant adaptation directions can be inferred directly from the pretrained weight geometry, without explicit reference to data-induced activation patterns.”

“decomposes pretrained weights into magnitude and direction components, utilizing LoRA for directional updates, reducing trainable parameters and enhancing fine-tuning performance, though its complexity and dependence on data quality may limit its effectiveness.”

“Although DoRA improves LoRA's learning capacity, its parameter count scales with the model's dimensionality since the magnitude component in DoRA is an n-dimensional trainable vector, where n represents the number of columns of the weight matrix.”

“This directly addresses a symptom of the scale ambiguity we identified, but it provides a heuristic fix without altering the core $BA^$ parameterization that creates the ambiguity.”

“DoRA liu2024dora and LoRA+ hayou2024lora, address limitations in LoRA's training dynamics”

“Learning parameter-based adaptation methods may struggle to generalize to out-of-distribution tasks, particularly when the injection of additional parameters is suboptimally placed, potentially leading to degraded performance”

“SoRA and DoRA both incur additional training overhead in the form of architectural modifications, importance calculations, additional regularization terms, or bespoke optimization strategies.”