Mixed-View Panorama Synthesis using Geospatially Guided Diffusion
This work addresses the challenge of uneven panorama coverage for panorama synthesis applications, though it appears incremental as it builds on existing diffusion and attention techniques for a new task setting.
The paper tackles the problem of synthesizing novel panoramas from a small set of input panoramas and a satellite image, addressing the mixed-view setting for arbitrary global locations. It introduces a diffusion-based method with an attention-based architecture, showing effectiveness in handling sparse or distant panorama inputs.
We introduce the task of mixed-view panorama synthesis, where the goal is to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synthesis for arbitrary locations worldwide. A critical challenge is that the spatial coverage of panoramas is uneven, with few panoramas available in many regions of the world. We introduce an approach that utilizes diffusion-based modeling and an attention-based architecture for extracting information from all available input imagery. Experimental results demonstrate the effectiveness of our proposed method. In particular, our model can handle scenarios when the available panoramas are sparse or far from the location of the panorama we are attempting to synthesize. The project page is available at https://mixed-view.github.io