Improving Multitask Retrieval by Promoting Task Specialization
This addresses the efficiency and performance gap in multitask retrieval for AI systems, though it is incremental as it builds on existing multitasking methods.
The paper tackled the problem of multitask retrieval underperforming compared to task-specific retrieval by promoting task specialization, resulting in a multitask retriever that outperforms task-specific retrievers on the KILT benchmark.
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval in which a separate retriever is trained for each task. We show that it is possible to train a multitask retriever that outperforms task-specific retrievers by promoting task specialization. The main ingredients are: (1) a better choice of pretrained model (one that is explicitly optimized for multitasking) along with compatible prompting, and (2) a novel adaptive learning method that encourages each parameter to specialize in a particular task. The resulting multitask retriever is highly performant on the KILT benchmark. Upon analysis, we find that the model indeed learns parameters that are more task-specialized compared to naive multitasking without prompting or adaptive learning.