PoF: Post-Training of Feature Extractor for Improving Generalization
This addresses generalization issues in deep learning for practitioners, but it is incremental as it builds on existing flatness-based methods.
The paper tackled the problem of improving generalization in deep models by developing PoF, a post-training algorithm that updates the feature extractor to find flatter minima, resulting in improved performance on CIFAR-10, CIFAR-100, and SVHN datasets with only 10-50 epochs of post-training.
It has been intensively investigated that the local shape, especially flatness, of the loss landscape near a minimum plays an important role for generalization of deep models. We developed a training algorithm called PoF: Post-Training of Feature Extractor that updates the feature extractor part of an already-trained deep model to search a flatter minimum. The characteristics are two-fold: 1) Feature extractor is trained under parameter perturbations in the higher-layer parameter space, based on observations that suggest flattening higher-layer parameter space, and 2) the perturbation range is determined in a data-driven manner aiming to reduce a part of test loss caused by the positive loss curvature. We provide a theoretical analysis that shows the proposed algorithm implicitly reduces the target Hessian components as well as the loss. Experimental results show that PoF improved model performance against baseline methods on both CIFAR-10 and CIFAR-100 datasets for only 10-epoch post-training, and on SVHN dataset for 50-epoch post-training. Source code is available at: \url{https://github.com/DensoITLab/PoF-v1