Improving Deep Regression with Tightness
This work addresses the problem of improving deep regression performance for machine learning practitioners and researchers working on regression tasks.
This work tackles the problem of improving deep regression by preserving ordinality, resulting in improved performance across various tasks, with the introduction of an optimal transport-based regularizer and a target duplication strategy. The effectiveness of these strategies is verified through experiments on three real-world regression tasks.
For deep regression, preserving the ordinality of the targets with respect to the feature representation improves performance across various tasks. However, a theoretical explanation for the benefits of ordinality is still lacking. This work reveals that preserving ordinality reduces the conditional entropy $H(Z|Y)$ of representation $Z$ conditional on the target $Y$. However, our findings reveal that typical regression losses do little to reduce $H(Z|Y)$, even though it is vital for generalization performance. With this motivation, we introduce an optimal transport-based regularizer to preserve the similarity relationships of targets in the feature space to reduce $H(Z|Y)$. Additionally, we introduce a simple yet efficient strategy of duplicating the regressor targets, also with the aim of reducing $H(Z|Y)$. Experiments on three real-world regression tasks verify the effectiveness of our strategies to improve deep regression. Code: https://github.com/needylove/Regression_tightness.