ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
This addresses limitations in image synthesis for applications like editing and inpainting, offering a novel hybrid approach that is incremental over existing autoregressive and diffusion methods.
The paper tackled the problem of unidirectional and single-scale context in autoregressive image synthesis by introducing ImageBART, which combines autoregressive modeling with a multinomial diffusion process for coarse-to-fine context incorporation. The result is improved image modification capabilities, such as free-form inpainting and text-guided editing without mask-specific training, while maintaining high-fidelity generation.
Autoregressive models and their sequential factorization of the data likelihood have recently demonstrated great potential for image representation and synthesis. Nevertheless, they incorporate image context in a linear 1D order by attending only to previously synthesized image patches above or to the left. Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete. It also processes the entire image on a single scale, thus ignoring more global contextual information up to the gist of the entire scene. As a remedy we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process: Whereas a multistage diffusion process successively removes information to coarsen an image, we train a (short) Markov chain to invert this process. In each stage, the resulting autoregressive ImageBART model progressively incorporates context from previous stages in a coarse-to-fine manner. Experiments show greatly improved image modification capabilities over autoregressive models while also providing high-fidelity image generation, both of which are enabled through efficient training in a compressed latent space. Specifically, our approach can take unrestricted, user-provided masks into account to perform local image editing. Thus, in contrast to pure autoregressive models, it can solve free-form image inpainting and, in the case of conditional models, local, text-guided image modification without requiring mask-specific training.