CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation
This addresses the need for better evaluation metrics in hardware design automation, though it is incremental as it builds on existing LLM-based code generation by adding a creativity dimension.
The paper tackles the problem of evaluating creativity in LLM-generated hardware code, which prior works overlooked by focusing only on functional correctness, and introduces CreativEval, a framework that quantifies four creative sub-components, finding GPT-3.5 as the most creative model.
Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs.