Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing
This addresses data efficiency and human-like iterative development in program synthesis, though it is incremental as it builds on existing bootstrapping and repairing methods.
The authors tackled the problem of limited dataset size and misaligned synthesis process in program synthesis language models by introducing a bootstrapping algorithm that teaches models to repair code, showing it outperforms regular fine-tuning and matches the performance of models 68% larger.
Language models for program synthesis are usually trained and evaluated on programming competition datasets (MBPP, APPS). However, these datasets are limited in size and quality, while these language models are extremely data hungry. Additionally, the language models have a misaligned program synthesis process compared to humans. While humans iteratively develop code with the help of a compiler, most program synthesis models currently produce code in one go. To solve these issues, we introduce a bootstrapping algorithm for program synthesis, that supports teaching models how to repair. We show that bootstrapping consistently outperforms regular fine-tuning. Compared to other work, our bootstrapped model performs on par with fine-tuned models that are 68\% larger. Notably, bootstrapping with repairing also improves non-repairing performance compared to regular bootstrapping during inference. However, on our models, repairing during inference is likely inferior to simply sampling the same number of solutions. Furthermore, we find that there are issues with the example test cases in the training portion of the APPS dataset that are valuable to the community, as many repairing and reinforcement learning methods rely on them.