Breaking hypothesis testing for failure rates
This work addresses the reliability of statistical tests in real-world applications, such as failure rate analysis in systems like Microsoft Azure, but it is incremental as it builds on existing tests without introducing a new paradigm.
The paper investigates the robustness of a Poisson point process rate test when its distributional assumptions are violated, finding that in some cases, such as with a Compound Poisson distribution, the test performs better with lower error rates, and it shows similar performance to a customized Wald test while being more general.
We describe the utility of point processes and failure rates and the most common point process for modeling failure rates, the Poisson point process. Next, we describe the uniformly most powerful test for comparing the rates of two Poisson point processes for a one-sided test (henceforth referred to as the "rate test"). A common argument against using this test is that real world data rarely follows the Poisson point process. We thus investigate what happens when the distributional assumptions of tests like these are violated and the test still applied. We find a non-pathological example (using the rate test on a Compound Poisson distribution with Binomial compounding) where violating the distributional assumptions of the rate test make it perform better (lower error rates). We also find that if we replace the distribution of the test statistic under the null hypothesis with any other arbitrary distribution, the performance of the test (described in terms of the false negative rate to false positive rate trade-off) remains exactly the same. Next, we compare the performance of the rate test to a version of the Wald test customized to the Negative Binomial point process and find it to perform very similarly while being much more general and versatile. Finally, we discuss the applications to Microsoft Azure. The code for all experiments performed is open source and linked in the introduction.