Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data
This work addresses the challenge of generalizing novel view synthesis to real-world scenes for applications like VR and robotics, though it is incremental in improving existing transformer methods.
The paper tackles the problem of limited dataset diversity in transformer-based novel view synthesis by incorporating synthetic data from diffusion models and introducing token disentanglement to reduce artifacts, achieving state-of-the-art results across benchmarks with reduced computational costs.
Large transformer-based models have made significant progress in generalizable novel view synthesis (NVS) from sparse input views, generating novel viewpoints without the need for test-time optimization. However, these models are constrained by the limited diversity of publicly available scene datasets, making most real-world (in-the-wild) scenes out-of-distribution. To overcome this, we incorporate synthetic training data generated from diffusion models, which improves generalization across unseen domains. While synthetic data offers scalability, we identify artifacts introduced during data generation as a key bottleneck affecting reconstruction quality. To address this, we propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning. This refinement not only improves reconstruction quality over standard transformers but also enables scalable training with synthetic data. As a result, our method outperforms existing models on both in-dataset and cross-dataset evaluations, achieving state-of-the-art results across multiple benchmarks while significantly reducing computational costs. Project page: https://scaling3dnvs.github.io/