Online Control of the False Discovery Rate under "Decision Deadlines"
This addresses the need for more flexible and adaptive statistical methods in sequential testing scenarios, such as in clinical trials or A/B testing, where immediate decisions are required but can be refined over time.
The paper tackles the problem of controlling false discoveries in online hypothesis testing by introducing a framework where preliminary decisions can be updated until a deadline, allowing for dynamic adjustments based on early results. It achieves control of the false discovery rate at every testing stage and under arbitrary dependency, with theoretical guarantees.
Online testing procedures aim to control the extent of false discoveries over a sequence of hypothesis tests, allowing for the possibility that early-stage test results influence the choice of hypotheses to be tested in later stages. Typically, online methods assume that a permanent decision regarding the current test (reject or not reject) must be made before advancing to the next test. We instead assume that each hypothesis requires an immediate preliminary decision, but also allows us to update that decision until a preset deadline. Roughly speaking, this lets us apply a Benjamini-Hochberg-type procedure over a moving window of hypotheses, where the threshold parameters for upcoming tests can be determined based on preliminary results. Our method controls the false discovery rate (FDR) at every stage of testing, as well as at adaptively chosen stopping times. These results apply even under arbitrary p-value dependency structures.