Robust Evidence for Declining Disruptiveness: Assessing the Role of Zero-Backward-Citation Works
This addresses a methodological debate in scientometrics about measuring scientific progress, but is incremental as it responds to a specific critique rather than introducing new findings.
The authors refute a critique that declining scientific disruptiveness is an artifact of including works with zero backward citations, showing that declines persist even with the critics' own data and methods, and identify severe data quality issues in the critique's dataset, including 20% non-research content.
We respond to Holst et al.'s critique that the decline in scientific disruptiveness documented in Park et al. (Nature, 2023) is an artifact of including works with zero backward citations. Using their advocated dataset, metric, and exclusion criteria, we find declines equivalent to major benchmark transformations in science. Their own regression model--designed to address their concerns about zero-citation works--yields large and significant declines for both papers and patents (p<0.001), a result found in their supplementary tables yet left unaddressed, despite directly contradicting their central claim. Their critique is further undermined by severe quality issues in their data, which contain three times more zero-citation works than ours. We trace this excess to their inclusion of at least 2.8 million editorials, obituaries, and comments, 1.5 million books and proceedings, and 254,000 product and artistic reviews--in all, 20% of their sample is non-research content that almost by definition lacks backward citations. Simple keyword searches confirm the problem's severity, identifying among others 456 For Dummies guides, 50 Dr. Seuss and Curious George books, and the Captain Underpants series--all zero-citation entries in their sample. Applying granular document type classification to their data reveals that such non-research content fell from 40% to 8% of their sample between 1945 and 2010--a shift sufficient to generate the decline in zero-citation prevalence they attribute to metadata errors in our study. Standard practice excludes such content to guard against the metadata quality concerns at the center of their critique--concerns their dataset exemplifies rather than addresses. Declining disruptiveness has been documented in nearly 100 studies across multiple databases, metrics, and non-citation-based measures. The weight of evidence does not support an artifact-based explanation.