PixelTransformer: Sample Conditioned Signal Generation
This addresses the need for flexible generative modeling in computer vision and beyond, though it appears incremental as it builds on existing generative frameworks.
The paper tackles the problem of generating spatial signals like images from sparse samples, proposing a generative model that infers distributions conditioned on arbitrary observed pixels, and shows it generates diverse samples with reduced variance as more pixels are observed across three image datasets.
We propose a generative model that can infer a distribution for the underlying spatial signal conditioned on sparse samples e.g. plausible images given a few observed pixels. In contrast to sequential autoregressive generative models, our model allows conditioning on arbitrary samples and can answer distributional queries for any location. We empirically validate our approach across three image datasets and show that we learn to generate diverse and meaningful samples, with the distribution variance reducing given more observed pixels. We also show that our approach is applicable beyond images and can allow generating other types of spatial outputs e.g. polynomials, 3D shapes, and videos.