Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words
This addresses a domain-specific issue for biomedical NLP applications, particularly in few-shot scenarios, but is incremental as it builds on existing prompt-based methods.
The paper tackles the problem of pre-trained models performing poorly on biomedical tasks due to rare biomedical words, especially in low-resource settings, by proposing a prompt-based fine-tuning approach that improves performance by up to 6% on a biomedical natural language inference task without extra parameters or training steps.
Prompt-based fine-tuning for pre-trained models has proven effective for many natural language processing tasks under few-shot settings in general domain. However, tuning with prompt in biomedical domain has not been investigated thoroughly. Biomedical words are often rare in general domain, but quite ubiquitous in biomedical contexts, which dramatically deteriorates the performance of pre-trained models on downstream biomedical applications even after fine-tuning, especially in low-resource scenarios. We propose a simple yet effective approach to helping models learn rare biomedical words during tuning with prompt. Experimental results show that our method can achieve up to 6% improvement in biomedical natural language inference task without any extra parameters or training steps using few-shot vanilla prompt settings.