Understanding Multi-View Transformers
This work addresses the challenge of improving and safely using multi-view transformers in 3D vision applications, but it is incremental as it focuses on analysis rather than new model development.
The paper tackles the problem of understanding the inner mechanisms of multi-view transformers like DUSt3R, which are black-box models in 3D vision, by probing and visualizing their residual connections to reveal how latent states develop across layers and how they differ from methods with explicit pose biases.
Multi-view transformers such as DUSt3R are revolutionizing 3D vision by solving 3D tasks in a feed-forward manner. However, contrary to previous optimization-based pipelines, the inner mechanisms of multi-view transformers are unclear. Their black-box nature makes further improvements beyond data scaling challenging and complicates usage in safety- and reliability-critical applications. Here, we present an approach for probing and visualizing 3D representations from the residual connections of the multi-view transformers' layers. In this manner, we investigate a variant of the DUSt3R model, shedding light on the development of its latent state across blocks, the role of the individual layers, and suggest how it differs from methods with stronger inductive biases of explicit global pose. Finally, we show that the investigated variant of DUSt3R estimates correspondences that are refined with reconstructed geometry. The code used for the analysis is available at https://github.com/JulienGaubil/und3rstand .