Diverse Image Inpainting with Bidirectional and Autoregressive Transformers
This addresses the challenge of limited perception fields in CNNs for image inpainting, offering a novel method that improves diversity and quality for applications like photo editing and restoration.
The paper tackles the problem of generating diverse and realistic content for image inpainting by proposing BAT-Fill, a framework that uses a bidirectional autoregressive transformer to model long-range dependencies and contextual information, achieving superior diversity and fidelity in experiments across multiple datasets.
Image inpainting is an underdetermined inverse problem, which naturally allows diverse contents to fill up the missing or corrupted regions realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in autoregressive transformers is suboptimal as corrupted image regions may have arbitrary shapes with contexts from any direction. We propose BAT-Fill, an innovative image inpainting framework that introduces a novel bidirectional autoregressive transformer (BAT) for image inpainting. BAT utilizes the transformers to learn autoregressive distributions, which naturally allows the diverse generation of missing contents. In addition, it incorporates the masked language model like BERT, which enables bidirectionally modeling of contextual information of missing regions for better image completion. Extensive experiments over multiple datasets show that BAT-Fill achieves superior diversity and fidelity in image inpainting qualitatively and quantitatively.