Transformable Bottleneck Networks
This addresses the problem of 3D image manipulation for computer vision and graphics applications, offering a novel method for spatial disentanglement in neural networks.
The paper tackles fine-grained 3D manipulation of image content by proposing Transformable Bottleneck Networks (TBNs), which apply spatial transformations to a volumetric bottleneck in an encoder-bottleneck-decoder architecture, achieving state-of-the-art results on novel view synthesis and enabling intuitive 3D image manipulations like non-rigid warping and single-image 3D reconstruction.
We propose a novel approach to performing fine-grained 3D manipulation of image content via a convolutional neural network, which we call the Transformable Bottleneck Network (TBN). It applies given spatial transformations directly to a volumetric bottleneck within our encoder-bottleneck-decoder architecture. Multi-view supervision encourages the network to learn to spatially disentangle the feature space within the bottleneck. The resulting spatial structure can be manipulated with arbitrary spatial transformations. We demonstrate the efficacy of TBNs for novel view synthesis, achieving state-of-the-art results on a challenging benchmark. We demonstrate that the bottlenecks produced by networks trained for this task contain meaningful spatial structure that allows us to intuitively perform a variety of image manipulations in 3D, well beyond the rigid transformations seen during training. These manipulations include non-uniform scaling, non-rigid warping, and combining content from different images. Finally, we extract explicit 3D structure from the bottleneck, performing impressive 3D reconstruction from a single input image.