AR$^2$: Adversarial Reinforcement Learning for Abstract Reasoning in Large Language Models
This addresses the challenge of improving generalization in coding-oriented LLMs by focusing on abstraction, though it appears incremental as it builds on existing reinforcement learning approaches.
The paper tackled the problem of enhancing abstraction abilities in large language models for code generation by proposing AR^2, a framework that uses adversarial reinforcement learning to train models on narrative-rich problems derived from kernel tasks, resulting in substantial accuracy improvements on unseen programming tasks.
Abstraction--the ability to recognize and distill essential computational patterns from complex problem statements--is a foundational skill in computer science, critical both for human problem-solvers and coding-oriented large language models (LLMs). Despite recent advances in training LLMs for code generation using reinforcement learning (RL), most existing approaches focus primarily on superficial pattern recognition, overlooking explicit training for abstraction. In this study, we propose AR$^2$ (Adversarial Reinforcement Learning for Abstract Reasoning), a novel framework explicitly designed to enhance the abstraction abilities of LLMs. AR$^2$ employs a teacher model to transform kernel problems into narrative-rich, challenging descriptions without changing their fundamental logic. Simultaneously, a student coding model is trained to solve these complex narrative problems by extracting their underlying computational kernels. Experimental results demonstrate that AR$^2$ substantially improves the student model's accuracy on previously unseen, challenging programming tasks, underscoring abstraction as a key skill for enhancing LLM generalization.