LoRA

“While effective for fine-tuning, existing LoRA-based methods face fundamental challenges when applied to pre-training from scratch. Unlike fine-tuning, where small adaptations naturally exhibit low-rank structure, pre-training from random initialization requires full-rank weight updates to learn diverse representations across the entire parameter space. This mismatch between LoRA's low-rank assumption and pre-training's full-rank requirements results in suboptimal performance in the pre-training stage.”

“Due to the design of MHSA, these modules include multiple heads (12 in ViT), and thus the weight updates approximated by the adapters also encompass updates for these multiple heads. However, is the pre-configured multi-head setup in Transformer necessarily essential?”

“While naive LoRA significantly degrades text performance, PLoRA can be seen to preserve the original text capabilities”

“Despite its success, LoRA and similar low-rank approaches still fall short of full fine-tuning in some settings.”

“Existing LoRA-style methods often assume a global low-rank structure across entire weight matrices. However, such assumptions can be overly restrictive: different tasks may activate different subspaces or require localized updates that a single low-rank component fails to capture.”

“However, the full potential of LoRA remains constrained by its inherent design limitations. Specifically, it assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across different modules and layers.”

“Yet, as discussed in Sec. sec: fur matrix based, this adaptation fails to capture the inherent complexity and spatial locality specific to convolution operations. The result is a reshaped two-dimensional structure that compromises the integrity of the original parameter space, leading to a representation that does not fully encapsulate the change of convolutional space.”

“a single LoRA module projects the features of different tasks into the same dense low-dimensional space, causing interference between tasks and failing to effectively separate the knowledge of different tasks”

“LoRA uniformly uses the same rank for all layers, without considering the difference across layers.”

“Despite its effectiveness and popularity, recent studies have underscored that LoRA and its variants face challenges such as diminishing performance~LoRA, and slower convergence~PiSSA relative to full fine-tuning, which deteriorate further as the rank declines~MoRA,HiRA.”

“However, its fixed-rank design across all layers limits flexibility and prevents adaptive capacity allocation to task-specific requirements.”

“Recognizing the LoRA's suboptimality of rigidly applying the same rank to all layers”