Learning from Contrastive Prompts: Automated Optimization and Adaptation
This addresses the need for automated and adaptable prompt engineering to reduce manual effort in deploying LLMs across varied contexts, representing a novel method for a known bottleneck.
The paper tackles the problem of manual prompt crafting for LLMs by proposing the Learning from Contrastive Prompts (LCP) framework, which achieves a win rate of over 76% over existing methods in prompt optimization on the Big-Bench Hard dataset and shows strong adaptability across model versions, families, and languages.
As LLMs evolve, significant effort is spent on manually crafting prompts. While existing prompt optimization methods automate this process, they rely solely on learning from incorrect samples, leading to a sub-optimal performance. Additionally, an unexplored challenge in the literature is prompts effective for prior models may not perform well on newer versions or different languages. We propose the Learning from Contrastive Prompts (LCP) framework to address these gaps, enhancing both prompt optimization and adaptation. LCP employs contrastive learning to generate effective prompts by analyzing patterns in good and bad prompt examples. Our evaluation on the Big-Bench Hard dataset shows that LCP has a win rate of over 76% over existing methods in prompt optimization and demonstrates strong adaptability across different model versions, families, and languages. LCP offers a systematic approach to prompt engineering, reducing manual effort in deploying LLMs across varied contexts.