Rethinking Prompt Optimization: Reinforcement, Diversification, and Migration in Blackbox LLMs
This work addresses the practical challenge of efficiently optimizing and migrating prompts for NLP applications using black-box LLM APIs, though it appears incremental in nature.
The paper tackles the problem of automatic prompt optimization for black-box LLMs by proposing a framework that enhances feedback mechanisms through positive reinforcement and diversification, and addresses prompt migration between models. Their approach achieved significant accuracy improvements, faster convergence, and lower computational costs compared to baselines.
An increasing number of NLP applications interact with large language models (LLMs) through black-box APIs, making prompt engineering critical for controlling model outputs. While recent Automatic Prompt Optimization (APO) methods iteratively refine prompts using model-generated feedback, textual gradients, they primarily focus on error correction and neglect valuable insights from correct predictions. This limits both their effectiveness and efficiency. In this paper, we propose a novel APO framework centered on enhancing the feedback mechanism. We reinterpret the textual gradient as a form of negative reinforcement and introduce the complementary positive reinforcement to explicitly preserve beneficial prompt components identified through successful predictions. To mitigate the noise inherent in LLM-generated feedback, we introduce a technique called feedback diversification, which aggregates multiple feedback signals, emphasizing consistent, actionable advice while filtering out outliers. Motivated by the rapid evolution and diversity of available LLMs, we also formalize Continual Prompt Optimization (CPO), addressing the practical challenge of efficiently migrating optimized prompts between different model versions or API providers. Our experiments reveal that naive prompt migration often degrades performance due to loss of critical instructions. In contrast, our approach consistently outperforms strong baselines, achieving significant accuracy improvements, faster convergence, and lower computational costs in both standard and migration scenarios.