Identifying Suitable Tasks for Inductive Transfer Through the Analysis of Feature Attributions
This addresses the computational inefficiency of brute-force search for effective transfer learning settings, which is a problem for researchers and practitioners in machine learning.
The paper tackles the problem of predicting whether transfer learning between tasks will be beneficial without performing experiments, by using explainability techniques to analyze neural network activations. The result shows a reduction in training time by up to 83.5% with only a minor performance cost of 0.034 F1 reduction on a specific dataset.
Transfer learning approaches have shown to significantly improve performance on downstream tasks. However, it is common for prior works to only report where transfer learning was beneficial, ignoring the significant trial-and-error required to find effective settings for transfer. Indeed, not all task combinations lead to performance benefits, and brute-force searching rapidly becomes computationally infeasible. Hence the question arises, can we predict whether transfer between two tasks will be beneficial without actually performing the experiment? In this paper, we leverage explainability techniques to effectively predict whether task pairs will be complementary, through comparison of neural network activation between single-task models. In this way, we can avoid grid-searches over all task and hyperparameter combinations, dramatically reducing the time needed to find effective task pairs. Our results show that, through this approach, it is possible to reduce training time by up to 83.5% at a cost of only 0.034 reduction in positive-class F1 on the TREC-IS 2020-A dataset.