Mapped Convolutions
This work addresses the limitation of standard convolutions to grid-based data, offering a method to expand CNNs to domains like spherical imaging and mesh processing, though it is incremental in adapting existing convolution concepts.
The paper tackles the problem of applying convolutional neural networks to non-grid structured data by introducing a 'mapped convolution' that decouples sampling and weighting, enabling operations on spherical data and meshes, resulting in a 17% improvement in spherical depth estimation state-of-the-art.
We present a versatile formulation of the convolution operation that we term a "mapped convolution." The standard convolution operation implicitly samples the pixel grid and computes a weighted sum. Our mapped convolution decouples these two components, freeing the operation from the confines of the image grid and allowing the kernel to process any type of structured data. As a test case, we demonstrate its use by applying it to dense inference on spherical data. We perform an in-depth study of existing spherical image convolution methods and propose an improved sampling method for equirectangular images. Then, we discuss the impact of data discretization when deriving a sampling function, highlighting drawbacks of the cube map representation for spherical data. Finally, we illustrate how mapped convolutions enable us to convolve directly on a mesh by projecting the spherical image onto a geodesic grid and training on the textured mesh. This method exceeds the state of the art for spherical depth estimation by nearly 17%. Our findings suggest that mapped convolutions can be instrumental in expanding the application scope of convolutional neural networks.