FFA-LoRA

“FFA-LoRA~sunimproving freezes one LoRA matrix to suppress cross-client interference. While effective, these methods trade off scalability, flexibility, or representational capacity.”

“However, using a fixed random matrix for $A$ limits the learning capability of LoRA, and we observe that optimizing only $B$ leads to significantly slower convergence.”

“Notably, combining direct weight aggregation with GaLore as an optimizer for local training steps significantly outperforms leading state-of-the-art LoRA methods like FlexLoRA and FFA-LoRA.”

“FFA-LoRA sun2024improving manages exact updates by freezing the $A$ parameter at initialization but reduces the model expressivity relative to vanilla LoRA.”

“However, we observe that with fewer finetuning parameters, FFA-LoRA is less robust than FedAVG of LoRA, primarily due to its reduced expressiveness from freezing down-projections.”

“Federated Freeze A LoRA (FFA-LoRA) sun2024improving mitigates this by keeping one set of adapters trainable, improving aggregation stability but limiting the training flexibility of other adapters.”

“However, freezing non-zero matrices will hinder the model from converging to a good local minimum, since random initialization is nearly impossible to produce optimal parameters for downstream tasks.”

“because certain matrices remain fixed, LoRA's capacity to adapt is limited, which often leading to suboptimal performance”

“However, we observe that with fewer fine-tuning parameters, FFA-LoRA is less robust than FedAVG for LoRA modules, primarily due to its limited expressiveness stemming from the restricted number of trainable parameters.”