MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark
This addresses a gap for researchers and practitioners in hardware design by providing a new benchmark, though it is incremental as it extends existing benchmarking efforts to a specialized domain.
The paper tackles the lack of benchmarks for hardware description languages in repository-level code completion by introducing MHRC-Bench, a multilingual benchmark covering three hardware design coding styles, and shows its effectiveness through comprehensive evaluation.
Large language models (LLMs) have achieved strong performance on code completion tasks in general-purpose programming languages. However, existing repository-level code completion benchmarks focus almost exclusively on software code and largely overlook hardware description languages. In this work, we present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark designed for multilingual hardware code completion at the repository level. Our benchmark targets completion tasks and covers three major hardware design coding styles. Each completion target is annotated with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree analysis. We conduct a comprehensive evaluation of models on MHRC-Bench-Eval. Comprehensive evaluation results and analysis demonstrate the effectiveness of MHRC-Bench.