MoFE: Mixture of Frozen Experts Architecture
This work addresses resource constraints in real-world environments by improving training efficiency for multi-domain models, though it is incremental as it builds on existing PEFT and MoE methods.
The paper tackles the problem of training efficiency and scalability in large models by proposing the Mixture of Frozen Experts (MoFE) architecture, which integrates Parameter-efficient Fine-tuning and Mixture of Experts to reduce trainable parameters, resulting in substantial efficiency gains with some performance trade-offs.
We propose the Mixture of Frozen Experts (MoFE) architecture, which integrates Parameter-efficient Fine-tuning (PEFT) and the Mixture of Experts (MoE) architecture to enhance both training efficiency and model scalability. By freezing the Feed Forward Network (FFN) layers within the MoE framework, MoFE significantly reduces the number of trainable parameters, improving training efficiency while still allowing for effective knowledge transfer from the expert models. This facilitates the creation of models proficient in multiple domains. We conduct experiments to evaluate the trade-offs between performance and efficiency, compare MoFE with other PEFT methodologies, assess the impact of domain expertise in the constituent models, and determine the optimal training strategy. The results show that, although there may be some trade-offs in performance, the efficiency gains are substantial, making MoFE a reasonable solution for real-world, resource-constrained environments.