Detecting Flakiness in Quantum Software: A Dynamic Testing Approach
For quantum software engineers and researchers, this provides the first empirical evidence on the prevalence and characteristics of flaky tests in quantum software, highlighting detection challenges under typical CI budgets.
This paper presents the first large-scale dynamic characterization of flaky tests in quantum software, analyzing Qiskit Terra across 23 releases with 10,000 executions each. They identified 62 flaky tests out of 27,026, with low overall flakiness rates (0-0.17%) but high recurrence (83.87% reappeared in multiple releases) and median failure probability of 9e-4, indicating detection requires thousands of executions.
Flaky tests, tests that pass or fail nondeterministically without changes to code or environment, pose a serious threat to software reliability. While classical software engineering has developed a rich body of techniques to study flakiness, corresponding evidence for quantum software remains limited. Prior work relies mainly on static analysis or small sets of manually reported incidents, leaving open questions about their prevalence, characteristics, and detectability. This paper presents the first large-scale dynamic characterization of flaky tests in quantum software, focusing on the Qiskit Terra core library. We executed the Qiskit Terra test suite 10,000 times across 23 releases in controlled environments. For each release, we measured test-outcome variability, identified flaky tests, estimated empirical failure probabilities, analyzed recurrence across versions, used Wilson confidence intervals to quantify rerun budgets for reliable detection, and mapped flaky tests to Terra subcomponents. Across 27,026 fully qualified test identifiers, we identified 62 unique flaky tests. Although overall flakiness rates were low (0-0.17%), recurrence was substantial: 52 of 62 flaky tests (83.87%) reappeared in multiple releases, while only 10 tests (16.13%) were confined to a single release. Empirical failure probabilities spanned several orders of magnitude, with a median of $9 \times 10^{-4}$ and 34 tests (54.84%) at or below $10^{-3}$, implying that thousands to tens of thousands of executions may be required for confident detection. These results show that quantum test flakiness is rare but difficult to detect under typical continuous integration budgets. To support future research, we release a public dataset of per-test execution outcomes.