LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Training
This addresses a specific bottleneck in training neural networks with auxiliary classifiers, offering a simple, hyperparameter-free solution for practitioners, though it is incremental as it builds on existing initialization methods.
The paper tackled the problem of weight initialization for deeply-supervised neural networks with auxiliary classifiers, where untrained auxiliary heads can destabilize early training, and proposed LION-DG, a layer-informed initialization method that accelerates convergence by up to 11.3% on CIFAR datasets while maintaining or improving accuracy.
Weight initialization remains decisive for neural network optimization, yet existing methods are largely layer-agnostic. We study initialization for deeply-supervised architectures with auxiliary classifiers, where untrained auxiliary heads can destabilize early training through gradient interference. We propose LION-DG, a layer-informed initialization that zero-initializes auxiliary classifier heads while applying standard He-initialization to the backbone. We prove that this implements Gradient Awakening: auxiliary gradients are exactly zero at initialization, then phase in naturally as weights grow -- providing an implicit warmup without hyperparameters. Experiments on CIFAR-10 and CIFAR-100 with DenseNet-DS and ResNet-DS architectures demonstrate: (1) DenseNet-DS: +8.3% faster convergence on CIFAR-10 with comparable accuracy, (2) Hybrid approach: Combining LSUV with LION-DG achieves best accuracy (81.92% on CIFAR-10), (3) ResNet-DS: Positive speedup on CIFAR-100 (+11.3%) with side-tap auxiliary design. We identify architecture-specific trade-offs and provide clear guidelines for practitioners. LION-DG is simple, requires zero hyperparameters, and adds no computational overhead.