Simulated Annealing in Early Layers Leads to Better Generalization
This work addresses generalization improvement for deep learning practitioners, but appears incremental as it modifies an existing state-of-the-art method (LLF) rather than introducing a fundamentally new approach.
The paper tackles the problem of improving neural network generalization by proposing SEAL, which applies simulated annealing to early layers instead of re-initializing later layers as in LLF. The result is significant performance gains over LLF on Tiny-ImageNet and transfer/few-shot learning tasks, with lower prediction depth indicating better average performance.
Recently, a number of iterative learning methods have been introduced to improve generalization. These typically rely on training for longer periods of time in exchange for improved generalization. LLF (later-layer-forgetting) is a state-of-the-art method in this category. It strengthens learning in early layers by periodically re-initializing the last few layers of the network. Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers. Essentially, later layers go through the normal gradient descent process, while the early layers go through short stints of gradient ascent followed by gradient descent. Extensive experiments on the popular Tiny-ImageNet dataset benchmark and a series of transfer learning and few-shot learning tasks show that we outperform LLF by a significant margin. We further show that, compared to normal training, LLF features, although improving on the target task, degrade the transfer learning performance across all datasets we explored. In comparison, our method outperforms LLF across the same target datasets by a large margin. We also show that the prediction depth of our method is significantly lower than that of LLF and normal training, indicating on average better prediction performance.