MURPHY: Multi-Turn GRPO for Self Correcting Code Generation
This work addresses the problem of enhancing reasoning capabilities for iterative tasks in AI, particularly in code generation, though it appears incremental as it builds on existing GRPO methods.
The paper tackled the challenge of improving large language models' performance on agentic tasks requiring iterative decision-making by introducing Murphy, a multi-turn reflective optimization framework that extends GRPO with self-correction, resulting in up to an 8% relative gain in pass@1 over GRPO on code generation benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful framework for enhancing the reasoning capabilities of large language models (LLMs). However, existing approaches such as Group Relative Policy Optimization (GRPO) and its variants, while effective on reasoning benchmarks, struggle with agentic tasks that require iterative decision-making. We introduce Murphy, a multi-turn reflective optimization framework that extends GRPO by incorporating iterative self-correction during training. By leveraging both quantitative and qualitative execution feedback, Murphy enables models to progressively refine their reasoning across multiple turns. Evaluations on code generation benchmarks with model families such as Qwen and OLMo show that Murphy consistently improves performance, achieving up to a 8% relative gain in pass@1 over GRPO, on similar compute budgets.