Faster Meta Update Strategy for Noise-Robust Deep Learning
This addresses the problem of slow training in meta-learning for researchers and practitioners dealing with noisy or biased data, representing an incremental improvement.
The paper tackles the slow training bottleneck in meta-learning for noise-robust deep learning by introducing a faster meta update strategy (FaMUS) that uses layer-wise approximation, saving two-thirds of training time while maintaining or improving generalization performance.
It has been shown that deep neural networks are prone to overfitting on biased training data. Towards addressing this issue, meta-learning employs a meta model for correcting the training bias. Despite the promising performances, super slow training is currently the bottleneck in the meta learning approaches. In this paper, we introduce a novel Faster Meta Update Strategy (FaMUS) to replace the most expensive step in the meta gradient computation with a faster layer-wise approximation. We empirically find that FaMUS yields not only a reasonably accurate but also a low-variance approximation of the meta gradient. We conduct extensive experiments to verify the proposed method on two tasks. We show our method is able to save two-thirds of the training time while still maintaining the comparable or achieving even better generalization performance. In particular, our method achieves the state-of-the-art performance on both synthetic and realistic noisy labels, and obtains promising performance on long-tailed recognition on standard benchmarks.