Patch Gradient Descent: Training Neural Networks on Very Large Images
This addresses a practical bottleneck for researchers and practitioners working with large-scale image data, offering an incremental improvement over standard gradient descent for memory-efficient training.
The paper tackles the problem of training CNN models on very large images, which is hindered by compute and memory constraints, by proposing Patch Gradient Descent (PatchGD), a method that updates models on small image patches iteratively, resulting in improved stability and efficiency, especially under limited memory conditions, as demonstrated on datasets like PANDA and UltraMNIST with ResNet50 and MobileNetV2.
Traditional CNN models are trained and tested on relatively low resolution images (<300 px), and cannot be directly operated on large-scale images due to compute and memory constraints. We propose Patch Gradient Descent (PatchGD), an effective learning strategy that allows to train the existing CNN architectures on large-scale images in an end-to-end manner. PatchGD is based on the hypothesis that instead of performing gradient-based updates on an entire image at once, it should be possible to achieve a good solution by performing model updates on only small parts of the image at a time, ensuring that the majority of it is covered over the course of iterations. PatchGD thus extensively enjoys better memory and compute efficiency when training models on large scale images. PatchGD is thoroughly evaluated on two datasets - PANDA and UltraMNIST with ResNet50 and MobileNetV2 models under different memory constraints. Our evaluation clearly shows that PatchGD is much more stable and efficient than the standard gradient-descent method in handling large images, and especially when the compute memory is limited.