ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Models
This work addresses the need for scalable and varied problem creation in programming education and AI testing, though it is incremental as it builds on existing generative models and intrinsic motivation concepts.
The authors tackled the problem of automating the generation of novel and challenging programming puzzles by proposing ACES, which uses autotelic generative models to produce diverse and difficult Python problems, resulting in puzzles that are three times more challenging than existing benchmarks on average across 11 state-of-the-art code LLMs.
The ability to invent novel and interesting problems is a remarkable feature of human intelligence that drives innovation, art, and science. We propose a method that aims to automate this process by harnessing the power of state-of-the-art generative models to produce a diversity of challenging yet solvable problems, here in the context of Python programming puzzles. Inspired by the intrinsically motivated literature, Autotelic CodE Search (ACES) jointly optimizes for the diversity and difficulty of generated problems. We represent problems in a space of LLM-generated semantic descriptors describing the programming skills required to solve them (e.g. string manipulation, dynamic programming, etc.) and measure their difficulty empirically as a linearly decreasing function of the success rate of Llama-3-70B, a state-of-the-art LLM problem solver. ACES iteratively prompts a large language model to generate difficult problems achieving a diversity of target semantic descriptors (goal-directed exploration) using previously generated problems as in-context examples. ACES generates problems that are more diverse and more challenging than problems produced by baseline methods and three times more challenging than problems found in existing Python programming benchmarks on average across 11 state-of-the-art code LLMs.