POLCA: Stochastic Generative Optimization with LLM
This work addresses the labor-intensive manual iteration required for optimizing complex AI systems, offering a scalable solution for researchers and practitioners in machine learning and AI, though it is incremental as it builds on existing optimization methods with novel adaptations.
The paper tackles the problem of optimizing complex systems like LLM prompts and multi-turn agents by formalizing it as a stochastic generative optimization problem, where a generative language model acts as the optimizer guided by rewards and feedback; it introduces POLCA, a scalable framework that achieves robust, sample- and time-efficient performance, consistently outperforming state-of-the-art algorithms on diverse benchmarks such as τ-bench, HotpotQA, VeriBench, and KernelBench.
Optimizing complex systems, ranging from LLM prompts to multi-turn agents, traditionally requires labor-intensive manual iteration. We formalize this challenge as a stochastic generative optimization problem where a generative language model acts as the optimizer, guided by numerical rewards and text feedback to discover the best system. We introduce Prioritized Optimization with Local Contextual Aggregation (POLCA), a scalable framework designed to handle stochasticity in optimization -- such as noisy feedback, sampling minibatches, and stochastic system behaviors -- while effectively managing the unconstrained expansion of solution space. POLCA maintains a priority queue to manage the exploration-exploitation tradeoff, systematically tracking candidate solutions and their evaluation histories. To enhance efficiency, we integrate an $\varepsilon$-Net mechanism to maintain parameter diversity and an LLM Summarizer to perform meta-learning across historical trials. We theoretically prove that POLCA converges to near-optimal candidate solutions under stochasticity. We evaluate our framework on diverse benchmarks, including $Ï$-bench, HotpotQA (agent optimization), VeriBench (code translation) and KernelBench (CUDA kernel generation). Experimental results demonstrate that POLCA achieves robust, sample and time-efficient performance, consistently outperforming state-of-the-art algorithms in both deterministic and stochastic problems. The codebase for this work is publicly available at https://github.com/rlx-lab/POLCA.