MaxCode: A Max-Reward Reinforcement Learning Framework for Automated Code Optimization
This addresses the problem of automating complex code optimization for developers, though it builds incrementally on existing search methods.
The paper tackles the challenge of using LLMs for code optimization by developing MaxCode, a max-reward reinforcement learning framework that integrates natural language critiques and generative reward-to-go models to guide iterative refinement based on execution feedback. It achieves 20.3% and 10.1% relative improvements in speedup metrics on CUDA and C++ benchmarks compared to baselines.
Large Language Models (LLMs) demonstrate strong capabilities in general coding tasks but encounter two key challenges when optimizing code: (i) the complexity of writing optimized code (such as performant CUDA kernels and competition-level CPU code) requires expertise in systems, algorithms and specific languages and (ii) requires interpretation of performance metrics like timing and device utilization beyond binary correctness. In this work, we explore inference-time search algorithms that guide the LLM to discover better solutions through iterative refinement based on execution feedback. Our approach, called MaxCode unifies existing search methods under a max-reward reinforcement learning framework, making the observation and action-value functions modular for modification. To enhance the observation space, we integrate a natural language critique model that converts raw execution feedback into diagnostic insights about errors and performance bottlenecks, and the best-discounted reward seen so far. Together, these provide richer input to the code proposal function. To improve exploration during search, we train a generative reward-to-go model using action values from rollouts to rerank potential solutions. Testing on the KernelBench (CUDA) and PIE (C++) optimization benchmarks shows that MaxCode improves optimized code performance compared to baselines, achieving 20.3% and 10.1% relative improvements in absolute speedup value and relative speedup ranking, respectively.