TEMPERA: Test-Time Prompting via Reinforcement Learning
This work addresses the challenge of optimizing prompts for zero-shot or few-shot learning in large language models, offering an adaptive and interpretable solution that improves efficiency for NLP practitioners.
The paper tackles the problem of automated prompt design for large language models by introducing TEMPERA, a test-time prompting method using reinforcement learning, which achieves a 5.33x average improvement in sample efficiency over traditional fine-tuning methods across tasks like sentiment analysis and natural language inference.
Careful prompt design is critical to the use of large language models in zero-shot or few-shot learning. As a consequence, there is a growing interest in automated methods to design optimal prompts. In this work, we propose Test-time Prompt Editing using Reinforcement learning (TEMPERA). In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge, is adaptive to different queries and provides an interpretable prompt for every query. To achieve this, we design a novel action space that allows flexible editing of the initial prompts covering a wide set of commonly-used components like instructions, few-shot exemplars, and verbalizers. The proposed method achieves significant gains compared with recent SoTA approaches like prompt tuning, AutoPrompt, and RLPrompt, across a variety of tasks including sentiment analysis, topic classification, natural language inference, and reading comprehension. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.