Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents
For practitioners using API-only LLM agents, AMC provides a principled method to optimize agent behavior at test time without parameter access, achieving strong results that scale with compute.
The authors propose Agentic Monte Carlo (AMC) to perform RL-style optimization of black-box LLM agents by directly sampling from the optimal policy via Sequential Monte Carlo, without modifying the underlying model. On three AgentGym environments, AMC significantly outperforms prompting baselines and even surpasses GRPO as test-time compute scales.
LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents. Code is available at https://github.com/layer6ai-labs/Agentic-Monte-Carlo