A distribution-free valid p-value for finite samples of bounded random variables
This work addresses calibration issues in machine learning and statistical inference, though it appears incremental as it builds on existing concentration inequalities and compares to prior methods.
The paper tackles the problem of constructing a valid p-value for bounded random variables in a distribution-free setting, motivated by calibrating predictive algorithms, and shows that their proposed p-value is tighter than Hoeffding and Bentkus alternatives in certain regions.
We build a valid p-value based on a concentration inequality for bounded random variables introduced by Pelekis, Ramon and Wang. The motivation behind this work is the calibration of predictive algorithms in a distribution-free setting. The super-uniform p-value is tighter than Hoeffding and Bentkus alternatives in certain regions. Even though we are motivated by a calibration setting in a machine learning context, the ideas presented in this work are also relevant in classical statistical inference. Furthermore, we compare the power of a collection of valid p- values for bounded losses, which are presented in previous literature.