DOMR: Establishing Cross-View Segmentation via Dense Object Matching
This addresses a critical challenge in visual understanding for applications like robotics and augmented reality, though it appears incremental as it builds on existing benchmarks and methods.
The paper tackles the problem of cross-view object correspondence between egocentric and exocentric views by proposing the DOMR framework, which achieves state-of-the-art performance with mean IoU scores of 49.7% on Ego→Exo and 55.2% on Exo→Ego, outperforming previous methods by 5.8% and 4.3% respectively.
Cross-view object correspondence involves matching objects between egocentric (first-person) and exocentric (third-person) views. It is a critical yet challenging task for visual understanding. In this work, we propose the Dense Object Matching and Refinement (DOMR) framework to establish dense object correspondences across views. The framework centers around the Dense Object Matcher (DOM) module, which jointly models multiple objects. Unlike methods that directly match individual object masks to image features, DOM leverages both positional and semantic relationships among objects to find correspondences. DOM integrates a proposal generation module with a dense matching module that jointly encodes visual, spatial, and semantic cues, explicitly constructing inter-object relationships to achieve dense matching among objects. Furthermore, we combine DOM with a mask refinement head designed to improve the completeness and accuracy of the predicted masks, forming the complete DOMR framework. Extensive evaluations on the Ego-Exo4D benchmark demonstrate that our approach achieves state-of-the-art performance with a mean IoU of 49.7% on Ego$\to$Exo and 55.2% on Exo$\to$Ego. These results outperform those of previous methods by 5.8% and 4.3%, respectively, validating the effectiveness of our integrated approach for cross-view understanding.