LVT: Large-Scale Scene Reconstruction via Local View Transformers
This addresses the problem of computational inefficiency in 3D vision for researchers and practitioners, representing an incremental improvement by optimizing existing transformer approaches for scalability.
The paper tackles the challenge of scaling transformer-based methods for large-scale 3D scene reconstruction and novel view synthesis by proposing the Local View Transformer (LVT), which avoids quadratic attention complexity and enables reconstruction of arbitrarily large, high-resolution scenes in a single forward pass.
Large transformer models are proving to be a powerful tool for 3D vision and novel view synthesis. However, the standard Transformer's well-known quadratic complexity makes it difficult to scale these methods to large scenes. To address this challenge, we propose the Local View Transformer (LVT), a large-scale scene reconstruction and novel view synthesis architecture that circumvents the need for the quadratic attention operation. Motivated by the insight that spatially nearby views provide more useful signal about the local scene composition than distant views, our model processes all information in a local neighborhood around each view. To attend to tokens in nearby views, we leverage a novel positional encoding that conditions on the relative geometric transformation between the query and nearby views. We decode the output of our model into a 3D Gaussian Splat scene representation that includes both color and opacity view-dependence. Taken together, the Local View Transformer enables reconstruction of arbitrarily large, high-resolution scenes in a single forward pass. See our project page for results and interactive demos https://toobaimt.github.io/lvt/.