Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions
This work addresses the labor-intensive labeling challenge for custom social-media analysis, though it is incremental as it builds on existing transfer and active learning methods.
The authors tackled the problem of reducing annotation effort for labeling social-media data on toxicity and bias by proposing an active transfer few-shot instructions (ATF) approach that avoids fine-tuning, achieving a mean AUC gain of 10.5% compared to no transfer with a large PLM.
Labeling social-media data for custom dimensions of toxicity and social bias is challenging and labor-intensive. Existing transfer and active learning approaches meant to reduce annotation effort require fine-tuning, which suffers from over-fitting to noise and can cause domain shift with small sample sizes. In this work, we propose a novel Active Transfer Few-shot Instructions (ATF) approach which requires no fine-tuning. ATF leverages the internal linguistic knowledge of pre-trained language models (PLMs) to facilitate the transfer of information from existing pre-labeled datasets (source-domain task) with minimum labeling effort on unlabeled target data (target-domain task). Our strategy can yield positive transfer achieving a mean AUC gain of 10.5% compared to no transfer with a large 22b parameter PLM. We further show that annotation of just a few target-domain samples via active learning can be beneficial for transfer, but the impact diminishes with more annotation effort (26% drop in gain between 100 and 2000 annotated examples). Finally, we find that not all transfer scenarios yield a positive gain, which seems related to the PLMs initial performance on the target-domain task.