Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?
This work addresses the challenge of designing effective prompts for zero-shot learning in NLP, offering insights and methods to improve prompt engineering, though it is incremental in advancing existing prompt tuning techniques.
The paper investigates what makes natural language prompts effective for zero-shot tasks with large language models, proposing a human-readable prompt tuning method that identifies topical relevance and label word calibration as key factors, and achieves a 7.0% average accuracy improvement over baselines across three tasks.
Large language models can perform new tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are natural language. In this paper, we investigate common attributes shared by effective prompts. We first propose a human readable prompt tuning method (F LUENT P ROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a diverse distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of label words. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks.