CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamics
This work addresses the need for standardized evaluation of LLMs in scientific computing, particularly for researchers and practitioners in CFD, though it is incremental as it builds on existing benchmarking practices.
The authors tackled the problem of evaluating large language models (LLMs) in automating numerical experiments for complex physical systems, specifically in computational fluid dynamics (CFD), by introducing CFDLLMBench, a benchmark suite that assesses LLM performance across knowledge, reasoning, and implementation tasks, with results quantified in terms of code executability, solution accuracy, and numerical convergence behavior.
Large Language Models (LLMs) have demonstrated strong performance across general NLP tasks, but their utility in automating numerical experiments of complex physical system -- a critical and labor-intensive component -- remains underexplored. As the major workhorse of computational science over the past decades, Computational Fluid Dynamics (CFD) offers a uniquely challenging testbed for evaluating the scientific capabilities of LLMs. We introduce CFDLLMBench, a benchmark suite comprising three complementary components -- CFDQuery, CFDCodeBench, and FoamBench -- designed to holistically evaluate LLM performance across three key competencies: graduate-level CFD knowledge, numerical and physical reasoning of CFD, and context-dependent implementation of CFD workflows. Grounded in real-world CFD practices, our benchmark combines a detailed task taxonomy with a rigorous evaluation framework to deliver reproducible results and quantify LLM performance across code executability, solution accuracy, and numerical convergence behavior. CFDLLMBench establishes a solid foundation for the development and evaluation of LLM-driven automation of numerical experiments for complex physical systems. Code and data are available at https://github.com/NREL-Theseus/cfdllmbench/.