Sequential Compression Layers for Efficient Federated Learning in Foundational Models
This addresses efficiency and performance bottlenecks in federated learning for foundational models, though it appears incremental as it modifies existing fine-tuning approaches.
The paper tackles the suboptimal performance of LoRA in federated fine-tuning of large models by proposing a novel parameter-efficient method that inserts a small MLP layer between existing layers in transformer blocks, achieving superior performance for language models and vision encoders.
Federated Learning (FL) has gained popularity for fine-tuning large language models (LLMs) across multiple nodes, each with its own private data. While LoRA has been widely adopted for parameter efficient federated fine-tuning, recent theoretical and empirical studies highlight its suboptimal performance in the federated learning context. In response, we propose a novel, simple, and more effective parameter-efficient fine-tuning method that does not rely on LoRA. Our approach introduces a small multi-layer perceptron (MLP) layer between two existing MLP layers the up proj (the FFN projection layer following the self-attention module) and down proj within the feed forward network of the transformer block. This solution addresses the bottlenecks associated with LoRA in federated fine tuning and outperforms recent LoRA-based approaches, demonstrating superior performance for both language models and vision encoders.