On Codex Prompt Engineering for OCL Generation: An Empirical Study
This addresses the challenge of OCL adoption due to unfamiliar syntax for software engineers, but it is incremental as it applies an existing method (Codex) to a new domain (OCL generation).
The study tackled the problem of generating Object Constraint Language (OCL) constraints from natural language specifications using Codex, finding that enriching prompts with UML information and using few-shot learning increased reliability, with results showing close similarity to human-written constraints based on sentence embedding.
The Object Constraint Language (OCL) is a declarative language that adds constraints and object query expressions to MOF models. Despite its potential to provide precision and conciseness to UML models, the unfamiliar syntax of OCL has hindered its adoption. Recent advancements in LLMs, such as GPT-3, have shown their capability in many NLP tasks, including semantic parsing and text generation. Codex, a GPT-3 descendant, has been fine-tuned on publicly available code from GitHub and can generate code in many programming languages. We investigate the reliability of OCL constraints generated by Codex from natural language specifications. To achieve this, we compiled a dataset of 15 UML models and 168 specifications and crafted a prompt template with slots to populate with UML information and the target task, using both zero- and few-shot learning methods. By measuring the syntactic validity and execution accuracy metrics of the generated OCL constraints, we found that enriching the prompts with UML information and enabling few-shot learning increases the reliability of the generated OCL constraints. Furthermore, the results reveal a close similarity based on sentence embedding between the generated OCL constraints and the human-written ones in the ground truth, implying a level of clarity and understandability in the generated OCL constraints by Codex.