CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation
This provides a new benchmark for researchers and practitioners in cloud computing and AI, addressing a gap in evaluating LLM-based code generation for cloud-native tools, though it is incremental as it builds on existing benchmarking efforts.
The paper tackles the lack of benchmarking for code generation in cloud-native applications by introducing CloudEval-YAML, a practical benchmark with 1011 hand-written problems that took over 1200 human hours to create, and it evaluates 12 LLMs to improve task performance and reduce cost.
Among the thriving ecosystem of cloud computing and the proliferation of Large Language Model (LLM)-based code generation tools, there is a lack of benchmarking for code generation in cloud-native applications. In response to this need, we present CloudEval-YAML, a practical benchmark for cloud configuration generation. CloudEval-YAML tackles the diversity challenge by focusing on YAML, the de facto standard of numerous cloud-native tools. We develop the CloudEval-YAML benchmark with practicality in mind: the dataset consists of hand-written problems with unit tests targeting practical scenarios. We further enhanced the dataset to meet practical needs by rephrasing questions in a concise, abbreviated, and bilingual manner. The dataset consists of 1011 problems that take more than 1200 human hours to complete. To improve practicality during evaluation, we build a scalable evaluation platform for CloudEval-YAML that achieves a 20 times speedup over a single machine. To the best of our knowledge, the CloudEval-YAML dataset is the first hand-written dataset targeting cloud-native applications. We present an in-depth evaluation of 12 LLMs, leading to a deeper understanding of the problems and LLMs, as well as effective methods to improve task performance and reduce cost.