Novel Bernstein-like Concentration Inequalities for the Missing Mass
This work addresses a fundamental issue in probability and learning theory by providing tighter bounds for the missing mass, though it is incremental as it builds on and refines existing methods.
The paper tackles the problem of deriving concentration inequalities for the missing mass, achieving distribution-free Bernstein-like bounds with sublinear exponents and improving prior results for small deviations, which are critical in learning theory.
We are concerned with obtaining novel concentration inequalities for the missing mass, i.e. the total probability mass of the outcomes not observed in the sample. We not only derive - for the first time - distribution-free Bernstein-like deviation bounds with sublinear exponents in deviation size for missing mass, but also improve the results of McAllester and Ortiz (2003) andBerend and Kontorovich (2013, 2012) for small deviations which is the most interesting case in learning theory. It is known that the majority of standard inequalities cannot be directly used to analyze heterogeneous sums i.e. sums whose terms have large difference in magnitude. Our generic and intuitive approach shows that the heterogeneity issue introduced in McAllester and Ortiz (2003) is resolvable at least in the case of missing mass via regulating the terms using our novel thresholding technique.