Asynchronous Online Testing of Multiple Hypotheses
This addresses the challenge of decentralized hypothesis testing in large organizations, offering a more realistic approach than classical methods that assume independence or arbitrary dependence, though it appears incremental in improving FDR control for specific scenarios.
The paper tackles the problem of controlling the false discovery rate (FDR) in asynchronous online testing, where tests can start and stop arbitrarily and dependencies exist, by introducing a framework using 'conflict sets' that provides formal FDR guarantees under local dependence, with simulations showing comparisons to existing algorithms.
We consider the problem of asynchronous online testing, aimed at providing control of the false discovery rate (FDR) during a continual stream of data collection and testing, where each test may be a sequential test that can start and stop at arbitrary times. This setting increasingly characterizes real-world applications in science and industry, where teams of researchers across large organizations may conduct tests of hypotheses in a decentralized manner. The overlap in time and space also tends to induce dependencies among test statistics, a challenge for classical methodology, which either assumes (overly optimistically) independence or (overly pessimistically) arbitrary dependence between test statistics. We present a general framework that addresses both of these issues via a unified computational abstraction that we refer to as "conflict sets." We show how this framework yields algorithms with formal FDR guarantees under a more intermediate, local notion of dependence. We illustrate our algorithms in simulations by comparing to existing algorithms for online FDR control.