On LLM-Assisted Generation of Smart Contracts from Business Processes
This addresses the problem of unreliable code generation for smart contracts from business processes, which is an incremental improvement over existing LLM-based methods by introducing automated evaluation.
The study investigated using large language models (LLMs) to generate smart contract code from business process descriptions, finding that LLM performance lacks the perfect reliability needed for smart contract development, with empirical data from larger datasets showing shortcomings in enforcing process flow, resource allocation, and data-based conditions.
Large language models (LLMs) have changed the reality of how software is produced. Within the wider software engineering community, among many other purposes, they are explored for code generation use cases from different types of input. In this work, we present an exploratory study to investigate the use of LLMs for generating smart contract code from business process descriptions, an idea that has emerged in recent literature to overcome the limitations of traditional rule-based code generation approaches. However, current LLM-based work evaluates generated code on small samples, relying on manual inspection, or testing whether code compiles but ignoring correct execution. With this work, we introduce an automated evaluation framework and provide empirical data from larger data sets of process models. We test LLMs of different types and sizes in their capabilities of achieving important properties of process execution, including enforcing process flow, resource allocation, and data-based conditions. Our results show that LLM performance falls short of the perfect reliability required for smart contract development. We suggest future work to explore responsible LLM integrations in existing tools for code generation to ensure more reliable output. Our benchmarking framework can serve as a foundation for developing and evaluating such integrations.