Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction
This work addresses the challenge of efficient and accurate video frame prediction for applications like video compression or real-time processing, but it is incremental as it compares existing architectures without introducing new methods.
The paper tackled the problem of video frame prediction by comparing feedforward and recurrent neural network architectures and training methods, finding that a residual fully convolutional neural network achieved the best PSNR but with higher computational cost, while a convolutional RNN offered near real-time performance with acceptable results.
We analyze the performance of feedforward vs. recurrent neural network (RNN) architectures and associated training methods for learned frame prediction. To this effect, we trained a residual fully convolutional neural network (FCNN), a convolutional RNN (CRNN), and a convolutional long short-term memory (CLSTM) network for next frame prediction using the mean square loss. We performed both stateless and stateful training for recurrent networks. Experimental results show that the residual FCNN architecture performs the best in terms of peak signal to noise ratio (PSNR) at the expense of higher training and test (inference) computational complexity. The CRNN can be trained stably and very efficiently using the stateful truncated backpropagation through time procedure, and it requires an order of magnitude less inference runtime to achieve near real-time frame prediction with an acceptable performance.