Glivenko-Cantelli for $f$-divergence
This work provides a foundational extension of a key statistical theorem, potentially impacting theoretical statistics and machine learning by enabling broader divergence-based analyses.
The authors tackled the problem of extending the Glivenko-Cantelli theorem from total variation distance to all f-divergences, achieving a novel integral representation of the Kolmogorov-Smirnov distance and establishing a Glivenko-Cantelli theorem for f-divergences on a π-system.
We extend the celebrated Glivenko-Cantelli theorem, sometimes called the fundamental theorem of statistics, from its standard setting of total variation distance to all $f$-divergences. A key obstacle in this endeavor is to define $f$-divergence on a subcollection of a $σ$-algebra that forms a $π$-system but not a $σ$-subalgebra. This is a side contribution of our work. We will show that this notion of $f$-divergence on the $π$-system of rays preserves nearly all known properties of standard $f$-divergence, yields a novel integral representation of the Kolmogorov-Smirnov distance, and has a Glivenko-Cantelli theorem. We will also discuss the prospects of a Vapnik-Chervonenkis theory for $f$-divergence.