ACING: Actor-Critic for Instruction Learning in Black-Box LLMs
This addresses the challenge of reducing human effort in crafting effective instructions for black-box LLMs, with incremental improvements in automated instruction optimization.
The paper tackles the problem of optimizing instructions for black-box large language models (LLMs) without access to gradients, introducing ACING, an actor-critic reinforcement learning framework that automatically discovers prompts outperforming human-written ones in 76% of tasks with gains up to 33 points and a 10-point median improvement over baselines.
The effectiveness of Large Language Models (LLMs) in solving tasks depends significantly on the quality of their instructions, which often require substantial human effort to craft. This underscores the need for automated instruction optimization. However, optimizing instructions is particularly challenging when working with black-box LLMs, where model parameters and gradients are inaccessible. We introduce ACING, an actor-critic reinforcement learning framework that formulates instruction optimization as a stateless, continuous-action problem, enabling exploration of infinite instruction spaces using only black-box feedback. ACING automatically discovers prompts that outperform human-written prompts in 76% of instruction-induction tasks, with gains of up to 33 points and a 10-point median improvement over the best automatic baseline in 33 tasks spanning instruction-induction, summarization, and chain-of-thought reasoning. Extensive ablations highlight its robustness and efficiency. An implementation of ACING is available at https://github.com/salmakh1/ACING.