ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions
This work addresses the challenge of high-resolution image editing for users in computer vision and graphics, offering a novel method that improves upon previous approaches, though it is incremental in advancing transformer-based techniques.
The authors tackled the problem of automatically editing high-resolution images based on semantic segmentation maps by introducing ASSET, a transformer-based architecture with a novel sparsified attention mechanism, which achieved efficient and effective synthesis of complex scene phenomena like reflections and consistent flora.
We present ASSET, a neural architecture for automatically modifying an input high-resolution image according to a user's edits on its semantic segmentation map. Our architecture is based on a transformer with a novel attention mechanism. Our key idea is to sparsify the transformer's attention matrix at high resolutions, guided by dense attention extracted at lower image resolutions. While previous attention mechanisms are computationally too expensive for handling high-resolution images or are overly constrained within specific image regions hampering long-range interactions, our novel attention mechanism is both computationally efficient and effective. Our sparsified attention mechanism is able to capture long-range interactions and context, leading to synthesizing interesting phenomena in scenes, such as reflections of landscapes onto water or flora consistent with the rest of the landscape, that were not possible to generate reliably with previous convnets and transformer approaches. We present qualitative and quantitative results, along with user studies, demonstrating the effectiveness of our method.