Learning Wake-Sleep Recurrent Attention Models
This addresses computational inefficiency in attention-based models for computer vision tasks, but appears incremental as it builds on existing deep generative training techniques.
The paper tackles the difficulty of training stochastic attention networks due to intractable posterior inference and high gradient variance, and presents the Wake-Sleep Recurrent Attention Model to improve training efficiency, showing it can greatly speed up training for image classification and caption generation.
Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.