What killed the Convex Booster ?
This clarifies a long-standing problem in machine learning theory, offering insights into model parameterization effects, though it is incremental in building on existing work.
The paper identifies the linear model class, not the convex loss or boosting algorithm, as the cause of a known failure in supervised learning, based on extending prior negative results and introducing a new boosting algorithm.
A landmark negative result of Long and Servedio established a worst-case spectacular failure of a supervised learning trio (loss, algorithm, model) otherwise praised for its high precision machinery. Hundreds of papers followed up on the two suspected culprits: the loss (for being convex) and/or the algorithm (for fitting a classical boosting blueprint). Here, we call to the half-century+ founding theory of losses for class probability estimation (properness), an extension of Long and Servedio's results and a new general boosting algorithm to demonstrate that the real culprit in their specific context was in fact the (linear) model class. We advocate for a more general stanpoint on the problem as we argue that the source of the negative result lies in the dark side of a pervasive -- and otherwise prized -- aspect of ML: \textit{parameterisation}.