Worst-Case-Aware Curriculum Learning for Zero and Few Shot Transfer
This addresses the issue of biased task representation in transfer learning for NLP applications, offering an incremental improvement over standard methods.
The paper tackles the problem of multi-task transfer learning by proposing a worst-case-aware curriculum learning approach to improve performance on outlier tasks and in zero-shot and few-shot transfer settings, resulting in better overall performance.
Multi-task transfer learning based on pre-trained language encoders achieves state-of-the-art performance across a range of tasks. Standard approaches implicitly assume the tasks, for which we have training data, are equally representative of the tasks we are interested in, an assumption which is often hard to justify. This paper presents a more agnostic approach to multi-task transfer learning, which uses automated curriculum learning to minimize a new family of worst-case-aware losses across tasks. Not only do these losses lead to better performance on outlier tasks; they also lead to better performance in zero-shot and few-shot transfer settings.