Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformers
This work addresses the computational bottleneck in visual geometric transformers, making them more practical for real-time applications like 3D perception and reconstruction, though it is incremental as it builds on existing acceleration techniques.
The paper tackles the problem of accelerating visual geometric transformers for real-time 3D perception by proposing Co-Me, a confidence-guided token merging method that reduces computation without retraining, achieving speedups of up to 11.3x and 7.2x on models like VGGT and MapAnything.
We propose Confidence-Guided Token Merging (Co-Me), an acceleration mechanism for visual geometric transformers without retraining or finetuning the base model. Co-Me distilled a light-weight confidence predictor to rank tokens by uncertainty and selectively merge low-confidence ones, effectively reducing computation while maintaining spatial coverage. Compared to similarity-based merging or pruning, the confidence signal in Co-Me reliably indicates regions emphasized by the transformer, enabling substantial acceleration without degrading performance. Co-Me applies seamlessly to various multi-view and streaming visual geometric transformers, achieving speedups that scale with sequence length. When applied to VGGT and MapAnything, Co-Me achieves up to $11.3\times$ and $7.2\times$ speedup, making visual geometric transformers practical for real-time 3D perception and reconstruction.