Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound
This provides a more information-rich generalization guarantee for machine learning models, especially useful in scenarios where error severity varies over time, though it is an incremental advance in PAC-Bayes theory.
The paper tackles the limitation of existing PAC-Bayes bounds to scalar performance metrics by introducing a new bound that controls the Kullback-Leibler divergence between empirical and true probabilities of multiple error types, enabling certificates for entire outcome distributions in regression and classification.
Current PAC-Bayes generalisation bounds are restricted to scalar metrics of performance, such as the loss or error rate. However, one ideally wants more information-rich certificates that control the entire distribution of possible outcomes, such as the distribution of the test loss in regression, or the probabilities of different mis-classifications. We provide the first PAC-Bayes bound capable of providing such rich information by bounding the Kullback-Leibler divergence between the empirical and true probabilities of a set of $M$ error types, which can either be discretized loss values for regression, or the elements of the confusion matrix (or a partition thereof) for classification. We transform our bound into a differentiable training objective. Our bound is especially useful in cases where the severity of different mis-classifications may change over time; existing PAC-Bayes bounds can only bound a particular pre-decided weighting of the error types. In contrast our bound implicitly controls all uncountably many weightings simultaneously.