Locally Supervised Learning with Periodic Global Guidance
This work addresses a key limitation in distributed neural network training for researchers and practitioners, though it is incremental as it builds on existing local learning methods.
The paper tackles the problem of poor generalization in locally supervised learning by introducing periodic global guidance, which significantly improves performance while maintaining low memory usage, achieving up to 15% accuracy gains on CIFAR-100 with ResNet-18.
Locally supervised learning aims to train a neural network based on a local estimation of the global loss function at each decoupled module of the network. Auxiliary networks are typically appended to the modules to approximate the gradient updates based on the greedy local losses. Despite being advantageous in terms of parallelism and reduced memory consumption, this paradigm of training severely degrades the generalization performance of neural networks. In this paper, we propose Periodically Guided local Learning (PGL), which reinstates the global objective repetitively into the local-loss based training of neural networks primarily to enhance the model's generalization capability. We show that a simple periodic guidance scheme begets significant performance gains while having a low memory footprint. We conduct extensive experiments on various datasets and networks to demonstrate the effectiveness of PGL, especially in the configuration with numerous decoupled modules.