An Information-Theoretic Approach to Transferability in Task Transfer Learning
This work addresses the challenge for practitioners in machine learning who need to efficiently select pre-trained models and plan transfer learning strategies, though it is incremental as it builds on existing transferability concepts with a new metric.
The paper tackles the problem of estimating task transferability in transfer learning by introducing a novel metric called H-score, which uses information-theoretic principles to predict how well representations from a source task help in learning a target task. Experiments on real image data show that the metric aligns with empirical measurements and is useful for applications like source model selection and curriculum learning.
Task transfer learning is a popular technique in image processing applications that uses pre-trained models to reduce the supervision cost of related tasks. An important question is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems using statistical and information theoretic principles. Experiments on real image data show that our metric is not only consistent with the empirical transferability measurement, but also useful to practitioners in applications such as source model selection and task transfer curriculum learning.