Grad Queue : A probabilistic framework to reinforce sparse gradients
This addresses the issue of gradient sparsity in large-batch training for machine learning practitioners, though it appears incremental as it builds on existing gradient optimization methods.
The paper tackles the problem of losing informative gradients in large batch updates by proposing a probabilistic framework that reinforces sparse gradients using a finite queue of online gradients and clustering to align objectives, resulting in superior performance on CIFAR10, MNIST, and Reuters News datasets compared to mini-batch gradient descent.
Informative gradients are often lost in large batch updates. We propose a robust mechanism to reinforce the sparse components within a random batch of data points. A finite queue of online gradients is used to determine their expected instantaneous statistics. We propose a function to measure the scarcity of incoming gradients using these statistics and establish the theoretical ground of this mechanism. To minimize conflicting components within large mini-batches, samples are grouped with aligned objectives by clustering based on inherent feature space. Sparsity is measured for each centroid and weighted accordingly. A strong intuitive criterion to squeeze out redundant information from each cluster is the backbone of the system. It makes rare information indifferent to aggressive momentum also exhibits superior performance with larger mini-batch horizon. The effective length of the queue kept variable to follow the local loss pattern. The contribution of our method is to restore intra-mini-batch diversity at the same time widening the optimal batch boundary. Both of these collectively drive it deeper towards the minima. Our method has shown superior performance for CIFAR10, MNIST, and Reuters News category dataset compared to mini-batch gradient descent.