PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models
This addresses the gap in evaluating LLMs for hardware-dependent tasks, particularly in physical computing education, though it is incremental as it establishes a new benchmark rather than a novel method.
The paper tackles the problem of evaluating large language models (LLMs) in physical computing by introducing PCEval, the first benchmark for automatic assessment, and finds that while LLMs perform well in code generation and logical circuit design, they struggle significantly with physical breadboard layout creation, with issues in pin connections and circuit errors.
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, including software development, education, and technical assistance. Among these, software development is one of the key areas where LLMs are increasingly adopted. However, when hardware constraints are considered-for instance, in physical computing, where software must interact with and control physical hardware -their effectiveness has not been fully explored. To address this gap, we introduce \textsc{PCEval} (Physical Computing Evaluation), the first benchmark in physical computing that enables a fully automatic evaluation of the capabilities of LLM in both the logical and physical aspects of the projects, without requiring human assessment. Our evaluation framework assesses LLMs in generating circuits and producing compatible code across varying levels of project complexity. Through comprehensive testing of 13 leading models, \textsc{PCEval} provides the first reproducible and automatically validated empirical assessment of LLMs' ability to reason about fundamental hardware implementation constraints within a simulation environment. Our findings reveal that while LLMs perform well in code generation and logical circuit design, they struggle significantly with physical breadboard layout creation, particularly in managing proper pin connections and avoiding circuit errors. \textsc{PCEval} advances our understanding of AI assistance in hardware-dependent computing environments and establishes a foundation for developing more effective tools to support physical computing education.