Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis
This work addresses the problem of slow and inefficient policy synthesis for control tasks by combining LLMs with gradient-free optimization, offering an incremental improvement in interpretable policy design.
The paper tackles the inefficiency of using large language models (LLMs) for generating symbolic control policies by decoupling structural synthesis from parameter optimization, resulting in higher returns and improved sample efficiency on control tasks compared to purely LLM-guided search.
Large Language models (LLMs) have shown promise as generators of symbolic control policies, producing interpretable program-like representations through iterative search. However, these models are not capable of separating the functional structure of a policy from the numerical values it is parametrized by, thus making the search process slow and inefficient. We propose a hybrid approach that decouples structural synthesis from parameter optimization by introducing an additional optimization layer for local parameter search. In our method, the numerical parameters of LLM-generated programs are extracted and optimized numerically to maximize task performance. With this integration, an LLM iterates over the functional structure of programs, while a separate optimization loop is used to find a locally optimal set of parameters accompanying candidate programs. We evaluate our method on a set of control tasks, showing that it achieves higher returns and improved sample efficiency compared to purely LLM-guided search. We show that combining symbolic program synthesis with numerical optimization yields interpretable yet high-performing policies, bridging the gap between language-model-guided design and classical control tuning. Our code is available at https://sites.google.com/berkeley.edu/colmo.