Pluto: A Benchmark for Evaluating Efficiency of LLM-generated Hardware Code
This addresses the need for efficiency-aware evaluation in hardware-focused LLM research, providing a domain-specific benchmark for researchers and practitioners.
The paper tackles the problem of evaluating the efficiency of LLM-generated Verilog code for hardware design, presenting Pluto as a benchmark with 114 problems and showing that state-of-the-art LLMs achieve 78.3% functional correctness but lag in synthesis efficiency with area, delay, and power efficiencies around 63-66%.
Large Language Models (LLMs) are increasingly used to automate hardware design tasks, including the generation of Verilog code. While early benchmarks focus primarily on functional correctness, efficient hardware design demands additional optimization for synthesis metrics such as area, delay, and power. Existing benchmarks fall short in evaluating these aspects comprehensively: they often lack optimized baselines or testbenches for verification. To address these gaps, we present Pluto, a benchmark and evaluation framework designed to assess the efficiency of LLM-generated Verilog designs. Pluto presents a comprehensive evaluation set of 114 problems with self-checking testbenches and multiple Pareto-optimal reference implementations. Experimental results show that state-of-the-art LLMs can achieve high functional correctness, reaching 78.3\% at pass@1, but their synthesis efficiency still lags behind expert-crafted implementations, with area efficiency of 63.8\%, delay efficiency of 65.9\%, and power efficiency of 64.0\% at eff@1. This highlights the need for efficiency-aware evaluation frameworks such as Pluto to drive progress in hardware-focused LLM research.