ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization
This addresses the problem of overfitting for practitioners using neural networks, offering a plug-and-play solution that is incremental but broadly applicable across architectures and tasks.
The paper tackles overfitting in neural networks by proposing ConsistentFeature, a regularization method that constrains feature differences across random subsets of the training set, resulting in reduced overfitting, improved accuracy, and lower validation loss with minimal computational cost.
Over-parameterized neural network models often lead to significant performance discrepancies between training and test sets, a phenomenon known as overfitting. To address this, researchers have proposed numerous regularization techniques tailored to various tasks and model architectures. In this paper, we introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets. Based on this viewpoint, we propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set. Due to minimal prior assumptions, this approach is applicable to almost any architecture and task. Our experiments show that it effectively reduces overfitting, with low sensitivity to hyperparameters and minimal computational cost. It demonstrates particularly strong memory suppression and promotes normal convergence, even when the model has already started to overfit. Even in the absence of significant overfitting, our method consistently improves accuracy and reduces validation loss.