Go Small and Similar: A Simple Output Decay Brings Better Performance
This addresses a gap in regularization methods for deep learning practitioners, offering a simple yet effective technique for enhancing model performance across various applications.
The paper tackles the problem of improving deep learning performance by regularizing model outputs, proposing Output Decay to enforce smaller and more similar output values, which results in remarkable performance gains as shown in extensive experiments.
Regularization and data augmentation methods have been widely used and become increasingly indispensable in deep learning training. Researchers who devote themselves to this have considered various possibilities. But so far, there has been little discussion about regularizing outputs of the model. This paper begins with empirical observations that better performances are significantly associated with output distributions, that have smaller average values and variances. By audaciously assuming there is causality involved, we propose a novel regularization term, called Output Decay, that enforces the model to assign smaller and similar output values on each class. Though being counter-intuitive, such a small modification result in a remarkable improvement on performance. Extensive experiments demonstrate the wide applicability, versatility, and compatibility of Output Decay.