Structured Program Synthesis using LLMs: Results and Insights from the IPARC Challenge
This work addresses the problem of automated program synthesis for researchers and practitioners, offering incremental insights into human-LLM collaboration mechanisms.
The paper tackled the IPARC Challenge, a set of 600 program synthesis tasks over synthetic images that had resisted automated solutions, and presented a structured inductive programming approach using LLMs that successfully solved tasks across all categories.
The IPARC Challenge, inspired by ARC, provides controlled program synthesis tasks over synthetic images to evaluate automatic program construction, focusing on sequence, selection, and iteration. This set of 600 tasks has resisted automated solutions. This paper presents a structured inductive programming approach with LLMs that successfully solves tasks across all IPARC categories. The controlled nature of IPARC reveals insights into LLM-based code generation, including the importance of prior structuring, LLMs' ability to aid structuring (requiring human refinement), the need to freeze correct code, the efficiency of code reuse, and how LLM-generated code can spark human creativity. These findings suggest valuable mechanisms for human-LLM collaboration in tackling complex program synthesis.