LLASP: Fine-tuning Large Language Models for Answer Set Programming
This work addresses a gap in applying LLMs to declarative programming formalisms like ASP, which is incremental as it adapts fine-tuning methods to a specific domain.
The paper tackles the problem of generating correct Answer Set Programming (ASP) code using large language models (LLMs), finding that existing LLMs perform inadequately, and proposes LLASP, a fine-tuned lightweight model that achieves remarkable quality in ASP program generation, outperforming non-fine-tuned and most eager LLM candidates.
Recently, Large Language Models (LLMs) have showcased their potential in various natural language processing tasks, including code generation. However, while significant progress has been made in adapting LLMs to generate code for several imperative programming languages and tasks, there remains a notable gap in their application to declarative formalisms, such as Answer Set Programming (ASP). In this paper, we move a step towards exploring the capabilities of LLMs for ASP code generation. First, we perform a systematic evaluation of several state-of-the-art LLMs. Despite their power in terms of number of parameters, training data and computational resources, empirical results demonstrate inadequate performances in generating correct ASP programs. Therefore, we propose LLASP, a fine-tuned lightweight model specifically trained to encode fundamental ASP program patterns. To this aim, we create an ad-hoc dataset covering a wide variety of fundamental problem specifications that can be encoded in ASP. Our experiments demonstrate that the quality of ASP programs generated by LLASP is remarkable. This holds true not only when compared to the non-fine-tuned counterpart but also when compared to the majority of eager LLM candidates, particularly from a semantic perspective. All the code and data used to perform the experiments are publicly available at https://anonymous.4open.science/r/LLASP-D86C/.