Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats
This addresses the problem of standardized evaluation for AI-assisted infrastructure management in cloud computing, though it is incremental as it builds on existing LLM capabilities with a new benchmark.
The paper tackles the lack of comprehensive benchmarks for evaluating Large Language Models (LLMs) in generating and modifying Infrastructure as Code (IaC) across multiple formats like AWS CloudFormation, Terraform, and CDK, by introducing Multi-IaC-Bench, a novel benchmark dataset. The result shows that modern LLMs achieve high success rates (>95%) in generating syntactically valid IaC but face challenges in semantic alignment and complex patterns.
Infrastructure as Code (IaC) is fundamental to modern cloud computing, enabling teams to define and manage infrastructure through machine-readable configuration files. However, different cloud service providers utilize diverse IaC formats. The lack of a standardized format requires cloud architects to be proficient in multiple IaC languages, adding complexity to cloud deployment. While Large Language Models (LLMs) show promise in automating IaC creation and maintenance, progress has been limited by the lack of comprehensive benchmarks across multiple IaC formats. We present Multi-IaC-Bench, a novel benchmark dataset for evaluating LLM-based IaC generation and mutation across AWS CloudFormation, Terraform, and Cloud Development Kit (CDK) formats. The dataset consists of triplets containing initial IaC templates, natural language modification requests, and corresponding updated templates, created through a synthetic data generation pipeline with rigorous validation. We evaluate several state-of-the-art LLMs on Multi-IaC-Bench, demonstrating that while modern LLMs can achieve high success rates (>95%) in generating syntactically valid IaC across formats, significant challenges remain in semantic alignment and handling complex infrastructure patterns. Our ablation studies highlight the importance of prompt engineering and retry mechanisms in successful IaC generation. We release Multi-IaC-Bench to facilitate further research in AI-assisted infrastructure management and establish standardized evaluation metrics for this crucial domain.