What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models
This work addresses the need for systematic comparison of process model representations to improve process modeling with large language models, but it is incremental as it focuses on empirical evaluation without introducing new methods.
The paper tackled the problem of comparing process model representations for large language model-based process modeling by evaluating nine representations on a new dataset of 55 process descriptions. It found that Mermaid scored highest overall for modeling suitability, while BPMN text performed best for process element similarity in generation tasks.
Large Language Models (LLMs) are increasingly applied for Process Modeling (PMo) tasks such as Process Model Generation (PMG). To support these tasks, researchers have introduced a variety of Process Model Representations (PMRs) that serve as model abstractions or generation targets. However, these PMRs differ widely in structure, complexity, and usability, and have never been systematically compared. Moreover, recent PMG approaches rely on distinct evaluation strategies and generation techniques, making comparison difficult. This paper presents the first empirical study that evaluates multiple PMRs in the context of PMo with LLMs. We introduce the PMo Dataset, a new dataset containing 55 process descriptions paired with models in nine different PMRs. We evaluate PMRs along two dimensions: suitability for LLM-based PMo and performance on PMG. \textit{Mermaid} achieves the highest overall score across six PMo criteria, whereas \textit{BPMN text} delivers the best PMG results in terms of process element similarity.