Balancing central and marginal rejection when combining independent significance tests
This work addresses a methodological issue in statistical hypothesis testing for researchers combining multiple tests, but it is incremental as it builds on existing pooling functions.
The paper tackles the problem of combining independent significance tests (p-values) into a single pooled p-value, focusing on balancing central and marginal rejection levels. It proposes a combining function based on the chi-square quantile transformation that controls this balance and is robust to parameter misspecification.
A common approach to evaluating the significance of a collection of $p$-values combines them with a pooling function, in particular when the original data are not available. These pooled $p$-values convert a sample of $p$-values into a single number which behaves like a univariate $p$-value. To clarify discussion of these functions, a telescoping series of alternative hypotheses are introduced that communicate the strength and prevalence of non-null evidence in the $p$-values before general pooling formulae are discussed. A pattern noticed in the UMP pooled $p$-value for a particular alternative motivates the definition and discussion of central and marginal rejection levels at $α$. It is proven that central rejection is always greater than or equal to marginal rejection, motivating a quotient to measure the balance between the two for pooled $p$-values. A combining function based on the $χ^2_κ$ quantile transformation is proposed to control this quotient and shown to be robust to mis-specified parameters relative to the UMP. Different powers for different parameter settings motivate a map of plausible alternatives based on where this pooled $p$-value is minimized.