A step towards understanding why classification helps regression
This addresses a practical problem for computer vision researchers and practitioners by clarifying when to use classification losses in regression tasks, though it is incremental in nature.
The paper investigates why adding a classification loss improves deep regression performance, finding it most beneficial for imbalanced data, and validates this on real datasets like NYUD2-DIR and IMDB-WIKI-DIR with concrete results.
A number of computer vision deep regression approaches report improved results when adding a classification loss to the regression loss. Here, we explore why this is useful in practice and when it is beneficial. To do so, we start from precisely controlled dataset variations and data samplings and find that the effect of adding a classification loss is the most pronounced for regression with imbalanced data. We explain these empirical findings by formalizing the relation between the balanced and imbalanced regression losses. Finally, we show that our findings hold on two real imbalanced image datasets for depth estimation (NYUD2-DIR), and age estimation (IMDB-WIKI-DIR), and on the problem of imbalanced video progress prediction (Breakfast). Our main takeaway is: for a regression task, if the data sampling is imbalanced, then add a classification loss.