OrchMoE: Efficient Multi-Adapter Learning with Task-Skill Synergy
This addresses multi-task learning efficiency for AI practitioners, representing an incremental advance in modular skill architectures.
The paper tackled the problem of Parameter-Efficient Fine-Tuning (PEFT) by introducing OrchMoE, a multi-adapter method that automatically identifies tasks and allocates skills, resulting in improved performance and sample efficiency on 1,600 instructional tasks compared to baselines.
We advance the field of Parameter-Efficient Fine-Tuning (PEFT) with our novel multi-adapter method, OrchMoE, which capitalizes on modular skill architecture for enhanced forward transfer in neural networks. Unlike prior models that depend on explicit task identification inputs, OrchMoE automatically discerns task categories, streamlining the learning process. This is achieved through an integrated mechanism comprising an Automatic Task Classification module and a Task-Skill Allocation module, which collectively deduce task-specific classifications and tailor skill allocation matrices. Our extensive evaluations on the 'Super Natural Instructions' dataset, featuring 1,600 diverse instructional tasks, indicate that OrchMoE substantially outperforms comparable multi-adapter baselines in terms of both performance and sample utilization efficiency, all while operating within the same parameter constraints. These findings suggest that OrchMoE offers a significant leap forward in multi-task learning efficiency.