Deep multi-scale video prediction beyond mean square error
This work addresses the challenge of generating clear future frames for video applications, offering an incremental improvement over existing methods.
The paper tackled the problem of blurry predictions in future video frame generation by proposing a convolutional network with multi-scale architecture, adversarial training, and an image gradient difference loss, achieving improved results on the UCF101 dataset compared to recurrent neural network methods.
Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics. This is why pixel-space video prediction may be viewed as a promising avenue for unsupervised feature learning. In addition, while optical flow has been a very studied problem in computer vision for a long time, future frame prediction is rarely approached. Still, many vision applications could benefit from the knowledge of the next frames of videos, that does not require the complexity of tracking every pixel trajectories. In this work, we train a convolutional network to generate future frames given an input sequence. To deal with the inherently blurry predictions obtained from the standard Mean Squared Error (MSE) loss function, we propose three different and complementary feature learning strategies: a multi-scale architecture, an adversarial training method, and an image gradient difference loss function. We compare our predictions to different published results based on recurrent neural networks on the UCF101 dataset