Synthetic Function Demonstrations Improve Generation in Low-Resource Programming Languages
This addresses the challenge of data scarcity for low-resource programming languages, offering a practical solution for developers and researchers, though it is incremental as it builds on existing finetuning and synthetic data methods.
The paper tackled the problem of training LLMs for low-resource programming languages by generating synthetic, textbook-quality demonstrations of common library functions in Excel formulas using a teacher model, and finetuning a student model, resulting in improved performance on two question-answering datasets recast into the Excel domain.
A key consideration when training an LLM is whether the target language is more or less resourced, whether this is English compared to Welsh, or Python compared to Excel. Typical training data for programming languages consist of real program demonstrations coupled with human-written comments. Here we present novel approaches to the creation of such data for low resource programming languages. We generate fully-synthetic, textbook-quality demonstrations of common library functions in an example domain of Excel formulas, using a teacher model. We then finetune an underperforming student model, and show improvement on 2 question-answering datasets recast into the Excel domain. We show advantages of finetuning over standard, off-the-shelf RAG approaches, which can offer only modest improvement due to the unfamiliar target domain.