OmniView: An All-Seeing Diffusion Model for 3D and 4D View Synthesis
This work addresses the need for a generalist model in 3D and 4D view synthesis for researchers and practitioners, though it is incremental as it builds on existing diffusion methods.
The paper tackles the problem of fragmented approaches for 4D consistency tasks in view synthesis by introducing OmniView, a unified diffusion model that generalizes across tasks like novel view synthesis and text-to-video with camera control, achieving improvements such as up to 33% higher image quality scores and 4x reduced camera trajectory errors.
Prior approaches injecting camera control into diffusion models have focused on specific subsets of 4D consistency tasks: novel view synthesis, text-to-video with camera control, image-to-video, amongst others. Therefore, these fragmented approaches are trained on disjoint slices of available 3D/4D data. We introduce OmniView, a unified framework that generalizes across a wide range of 4D consistency tasks. Our method separately represents space, time, and view conditions, enabling flexible combinations of these inputs. For example, OmniView can synthesize novel views from static, dynamic, and multiview inputs, extrapolate trajectories forward and backward in time, and create videos from text or image prompts with full camera control. OmniView is competitive with task-specific models across diverse benchmarks and metrics, improving image quality scores among camera-conditioned diffusion models by up to 33\% in multiview NVS LLFF dataset, 60\% in dynamic NVS Neural 3D Video benchmark, 20\% in static camera control on RE-10K, and reducing camera trajectory errors by 4x in text-conditioned video generation. With strong generalizability in one model, OmniView demonstrates the feasibility of a generalist 4D video model. Project page is available at https://snap-research.github.io/OmniView/