Nataliia Klievtsova

Nataliia Klievtsova, Janik-Vasily Benzin, Timotheus Kampik et al.

AI-driven chatbots such as ChatGPT have caused a tremendous hype lately. For BPM applications, several applications for AI-driven chatbots have been identified to be promising to generate business value, including explanation of process mining outcomes and preparation of input data. However, a systematic analysis of chatbots for their support of conversational process modeling as a process-oriented capability is missing. This work aims at closing this gap by providing a systematic analysis of existing chatbots. Application scenarios are identified along the process life cycle. Then a systematic literature review on conversational process modeling is performed, resulting in a taxonomy of application scenarios for conversational process modeling, including paraphrasing and improvement of process descriptions. In addition, this work suggests and applies an evaluation method for the output of AI-driven chatbots with respect to completeness and correctness of the process models. This method consists of a set of KPIs on a test set, a set of prompts for task and control flow extraction, as well as a survey with users. Based on the literature and the evaluation, recommendations for the usage (practical implications) and further development (research directions) of conversational process modeling are derived.

Nataliia Klievtsova, Timotheus Kampik, Juergen Mangler et al.

With the recent success of large language models (LLMs), the idea of AI-augmented Business Process Management systems is becoming more feasible. One of their essential characteristics is the ability to be conversationally actionable, allowing humans to interact with the LLM effectively to perform crucial process life cycle tasks such as process model design and redesign. However, most current research focuses on single-prompt execution and evaluation of results, rather than on continuous interaction between the user and the LLM. In this work, we aim to explore the feasibility of using LLMs to empower domain experts in the creation and redesign of process models in an iterative and effective way. The proposed conversational process model redesign (CPD) approach receives as input a process model and a redesign request by the user in natural language. Instead of just letting the LLM make changes, the LLM is employed to (a) identify process change patterns from literature, (b) re-phrase the change request to be aligned with an expected wording for the identified pattern (i.e., the meaning), and then to (c) apply the meaning of the change to the process model. This multi-step approach allows for explainable and reproducible changes. In order to ensure the feasibility of the CPD approach, and to find out how well the patterns from literature can be handled by the LLM, we performed an extensive evaluation. The results show that some patterns are hard to understand by LLMs and by users. Within the scope of the study, we demonstrated that users need support to describe the changes clearly. Overall the evaluation shows that the LLMs can handle most changes well according to a set of completeness and correctness criteria.