Code Aesthetics with Agentic Reward Feedback
This addresses the specific problem of visually-oriented coding tasks for developers using LLM assistants, representing a domain-specific incremental improvement.
The paper tackles the problem of poor aesthetic quality in LLM-generated code by introducing a pipeline that combines supervised fine-tuning on a new 358K instruction dataset with reinforcement learning using multi-agent reward feedback. Their 4B parameter model surpasses GPT-4o and GPT-4.1, achieving performance comparable to much larger 480B-685B open-source models on code aesthetics benchmarks.
Large Language Models (LLMs) have become valuable assistants for developers in code-related tasks. While LLMs excel at traditional programming tasks such as code generation and bug fixing, they struggle with visually-oriented coding tasks, often producing suboptimal aesthetics. In this paper, we introduce a new pipeline to enhance the aesthetic quality of LLM-generated code. We first construct AesCode-358K, a large-scale instruction-tuning dataset focused on code aesthetics. Next, we propose agentic reward feedback, a multi-agent system that evaluates executability, static aesthetics, and interactive aesthetics. Building on this, we develop GRPO-AR, which integrates these signals into the GRPO algorithm for joint optimization of functionality and code aesthetics. Finally, we develop OpenDesign, a benchmark for assessing code aesthetics. Experimental results show that combining supervised fine-tuning on AesCode-358K with reinforcement learning using agentic reward feedback significantly improves performance on OpenDesign and also enhances results on existing benchmarks such as PandasPlotBench. Notably, our AesCoder-4B surpasses GPT-4o and GPT-4.1, and achieves performance comparable to large open-source models with 480B-685B parameters, underscoring the effectiveness of our approach.