Decision-Theoretic Stopping Rules for Document Screening
For professionals conducting document screening (e.g., patent examiners, systematic reviewers), this provides a principled way to stop reviewing earlier or later based on cost-benefit trade-offs.
The paper applies decision theory to derive stopping rules for document screening that maximize net utility, outperforming existing recall-based methods on patent and systematic review datasets.
Deciding when to stop reviewing the results of a search is a common problem with multiple applications. Existing stopping rules developed within Technology-Assisted Review (TAR) aim to achieve a pre-specified recall target and do not take into account the reason for examining the results, potentially leading to sub-optimal recommendations. This paper applies decision theory to the problem and uses it to derive three practical stopping policies based on the Expected Value of Perfect Information. The approach is applied to two professional search tasks: patent examining and systematic reviewing. Experiments on CLEF-IP and medical systematic review datasets show that the proposed approach generally produces more appropriate stopping decisions than existing methods, as demonstrated by higher net utility under the evaluated cost and payoff settings.