Learning Spherical Convolution for Fast Features from 360° Imagery
This addresses the challenge of efficient and accurate feature extraction for 360° images and video, enabling the use of pre-trained networks in applications like vision, graphics, and augmented reality, but it is incremental as it adapts existing methods to a specific domain.
The paper tackles the problem of feature extraction from 360° imagery by proposing a spherical convolutional network that translates planar CNNs to process equirectangular projections directly, resulting in accurate results with orders of magnitude computational savings compared to existing methods.
While 360° cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make core feature extraction non-trivial. Convolutional neural networks (CNNs) trained on images from perspective cameras yield "flat" filters, yet 360° images cannot be projected to a single plane without significant distortion. A naive solution that repeatedly projects the viewing sphere to all tangent planes is accurate, but much too computationally intensive for real problems. We propose to learn a spherical convolutional network that translates a planar CNN to process 360° imagery directly in its equirectangular projection. Our approach learns to reproduce the flat filter outputs on 360° data, sensitive to the varying distortion effects across the viewing sphere. The key benefits are 1) efficient feature extraction for 360° images and video, and 2) the ability to leverage powerful pre-trained networks researchers have carefully honed (together with massive labeled image training sets) for perspective images. We validate our approach compared to several alternative methods in terms of both raw CNN output accuracy as well as applying a state-of-the-art "flat" object detector to 360° data. Our method yields the most accurate results while saving orders of magnitude in computation versus the existing exact reprojection solution.