Optimistic bounds for multi-output prediction
This work addresses multi-output prediction problems like multi-target regression and classification, providing theoretical guarantees that are incremental improvements over existing bounds.
The paper tackles the challenge of multi-output learning by introducing a self-bounding Lipschitz condition for loss functions, which leads to optimistic generalization bounds that are minimax optimal up to logarithmic factors, and applies this to derive a state-of-the-art bound for multi-class gradient boosting.
We investigate the challenge of multi-output learning, where the goal is to learn a vector-valued function based on a supervised data set. This includes a range of important problems in Machine Learning including multi-target regression, multi-class classification and multi-label classification. We begin our analysis by introducing the self-bounding Lipschitz condition for multi-output loss functions, which interpolates continuously between a classical Lipschitz condition and a multi-dimensional analogue of a smoothness condition. We then show that the self-bounding Lipschitz condition gives rise to optimistic bounds for multi-output learning, which are minimax optimal up to logarithmic factors. The proof exploits local Rademacher complexity combined with a powerful minoration inequality due to Srebro, Sridharan and Tewari. As an application we derive a state-of-the-art generalization bound for multi-class gradient boosting.