Train on Validation: Squeezing the Data Lemon
This addresses the practical issue of data scarcity in machine learning for practitioners, though it is incremental by building on existing stable algorithms.
The paper tackles the problem of model selection overfitting by proposing a method to use validation data for training, allowing a controlled trade-off between performance and overfitting. Results on MNIST and CIFAR-10 show a significant increase in test performance with minor bias.
Model selection on validation data is an essential step in machine learning. While the mixing of data between training and validation is considered taboo, practitioners often violate it to increase performance. Here, we offer a simple, practical method for using the validation set for training, which allows for a continuous, controlled trade-off between performance and overfitting of model selection. We define the notion of on-average-validation-stable algorithms as one in which using small portions of validation data for training does not overfit the model selection process. We then prove that stable algorithms are also validation stable. Finally, we demonstrate our method on the MNIST and CIFAR-10 datasets using stable algorithms as well as state-of-the-art neural networks. Our results show significant increase in test performance with a minor trade-off in bias admitted to the model selection process.