Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
This work addresses the challenge of enhancing LLM performance efficiently for task-agnostic learning, though it appears incremental as it builds on existing MoE and instruction tuning techniques.
The paper tackles the problem of improving large language models (LLMs) by combining Mixture-of-Experts (MoE) architecture with instruction tuning, finding that MoE models benefit more from this combination than dense models, with FLAN-MOE-32B outperforming FLAN-PALM-62B on four benchmark tasks while using only a third of the FLOPs.
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost. Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instructiontuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (second and third scenario), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied byFLAN-MOE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.