Authoring image decompositions with generative models
This work addresses the challenge of multi-layer image decomposition for computer vision researchers, offering a general method that does not rely on physical interpretations, though it is incremental in building on existing generative model techniques.
The authors tackled the problem of extending intrinsic image decompositions to multiple layers by learning to decompose images into layers using generative models for each layer, achieving high fidelity reconstructions with a novel Convolutional Variational Auto Encoder architecture.
We show how to extend traditional intrinsic image decompositions to incorporate further layers above albedo and shading. It is hard to obtain data to learn a multi-layer decomposition. Instead, we can learn to decompose an image into layers that are "like this" by authoring generative models for each layer using proxy examples that capture the Platonic ideal (Mondrian images for albedo; rendered 3D primitives for shading; material swatches for shading detail). Our method then generates image layers, one from each model, that explain the image. Our approach rests on innovation in generative models for images. We introduce a Convolutional Variational Auto Encoder (conv-VAE), a novel VAE architecture that can reconstruct high fidelity images. The approach is general, and does not require that layers admit a physical interpretation.