Deep 360$^\circ$ Optical Flow Estimation Based on Multi-Projection Fusion
This addresses a less explored problem in video processing for VR applications, but it is incremental as it builds on existing deep learning methods for optical flow.
The paper tackled 360° optical flow estimation for VR applications by proposing a multi-projection fusion framework to address distortions in panoramic representations, and it built the first large-scale dataset, outperforming existing methods on this dataset.
Optical flow computation is essential in the early stages of the video processing pipeline. This paper focuses on a less explored problem in this area, the 360$^\circ$ optical flow estimation using deep neural networks to support increasingly popular VR applications. To address the distortions of panoramic representations when applying convolutional neural networks, we propose a novel multi-projection fusion framework that fuses the optical flow predicted by the models trained using different projection methods. It learns to combine the complementary information in the optical flow results under different projections. We also build the first large-scale panoramic optical flow dataset to support the training of neural networks and the evaluation of panoramic optical flow estimation methods. The experimental results on our dataset demonstrate that our method outperforms the existing methods and other alternative deep networks that were developed for processing 360° content.