Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models
This work addresses the problem of template sensitivity in zero-shot learning for NLP practitioners, offering a more flexible and interpretable alternative, though it is incremental as it builds on existing prompting methods.
The paper tackles the sensitivity of zero-shot text classification to prompt templates by proposing a mining-based approach that uses regular expressions to retrieve labeled examples from unlabeled corpora, optionally filtered through prompting, and finetunes a pretrained model, outperforming prompting on a wide range of tasks with comparable templates.
Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.