D.Va: Validate Your Demonstration First Before You Use It
This addresses the challenge of demonstration selection for researchers and practitioners using in-context learning, offering a more reliable approach compared to intuitive metrics.
The paper tackles the problem of selecting effective demonstrations for in-context learning in large language models by proposing D.Va, a method that validates demonstrations to improve robustness and generalization, achieving state-of-the-art results across NLU and NLG tasks.
In-context learning (ICL) has demonstrated significant potential in enhancing the capabilities of large language models (LLMs) during inference. It's well-established that ICL heavily relies on selecting effective demonstrations to generate outputs that better align with the expected results. As for demonstration selection, previous approaches have typically relied on intuitive metrics to evaluate the effectiveness of demonstrations, which often results in limited robustness and poor cross-model generalization capabilities. To tackle these challenges, we propose a novel method, \textbf{D}emonstration \textbf{VA}lidation (\textbf{D.Va}), which integrates a demonstration validation perspective into this field. By introducing the demonstration validation mechanism, our method effectively identifies demonstrations that are both effective and highly generalizable. \textbf{D.Va} surpasses all existing demonstration selection techniques across both natural language understanding (NLU) and natural language generation (NLG) tasks. Additionally, we demonstrate the robustness and generalizability of our approach across various language models with different retrieval models.