Benchmarking Quantum Computers via Protocols, Comparing IBM's Heron vs IBM's Eagle
This work provides a transparent assessment of quantum processors for researchers and developers, though it is incremental as it extends an existing method to new hardware.
The paper tackled the challenge of objectively evaluating quantum computing hardware by applying a protocol-based benchmarking methodology to compare IBM's Eagle and Heron architectures, revealing substantial performance improvements in the newer Heron generation.
As quantum computing hardware rapidly advances, objectively evaluating the capabilities and error rates of new processors remains a critical challenge for the field. A clear and realistic understanding of current quantum performance is essential to guide research priorities and drive meaningful progress. In this work, we apply and extend a protocol-based benchmarking methodology (presented in arXiv:2505.12441) that utilizes well-defined quantumness thresholds. By evaluating performance at protocol level rather then the gate level, this approach provides a transparent and intuitive assessment of whether specific quantum processors, or isolated sub-chips within them, can demonstrate a practical quantum advantage. To illustrate the utility of this method, we compare two generations of IBM quantum computers: the older Eagle architecture and the newer Heron architecture. Our findings reveal the genuine operational strengths and limitations of these devices, demonstrating substantial performance improvements in the newer Heron generation.