Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning
This addresses the critical but inconsistent task selection problem in intermediate-task transfer learning for NLP practitioners, though it is incremental in improving existing methods.
The paper tackles the problem of selecting beneficial intermediate tasks for transfer learning by experimenting with 130 source-target combinations, showing that transfer performance varies significantly across tasks and seeds. It finds that task embeddings from fine-tuned weights improve task prediction scores from 2.59% to 3.96%, and a novel token similarity method achieves the highest performance.
Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. In this work, we experiment with 130 source-target task combinations and demonstrate that the transfer performance exhibits severe variance across different source tasks and training seeds, highlighting the crucial role of intermediate-task selection in a broader context. We compare four representative task selection methods in a unified setup, focusing on their effectiveness and consistency. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability by improving task prediction scores from 2.59% to 3.96%. Despite their strong performance, we observe that the task embeddings do not consistently demonstrate superiority for tasks requiring reasoning abilities. Furthermore, we introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction. Our findings suggest that token-wise similarity is better predictive for predicting transferability compared to averaging weights.