Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
This addresses the challenge for scientists and developers in creating complex data analysis pipelines, though it is incremental in applying existing LLM technology to a specific domain.
The study tackled the difficulty of implementing scientific workflows by evaluating ChatGPT's efficiency in supporting users, finding that it effectively interprets workflows but performs poorly for component exchange or purposeful extensions.
Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages. To address these challenges, we investigate the efficiency of Large Language Models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed three user studies in two scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.