Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers
This reveals an emergent capability in 2D detection transformers that could be leveraged for 3D tasks without additional supervision, relevant to computer vision researchers.
The paper probes pre-trained 2D detection transformers (DETR and variants) to see if their object-level embeddings encode 3D properties like depth and 3D location. They find that these models, despite no 3D supervision, surprisingly encode useful 3D information, recoverable with linear and non-linear probes.
Detection transformer models, including DETR and its extensions, learn to output a set of object-level embeddings that can be simultaneously decoded into 2D bounding boxes and class distributions. In this paper, we investigate what pre-trained 2D detection transformers understand about the 3D properties of objects. Specifically, we investigate the extent to which properties including the depth of objects from the camera and the 3D location of objects relative to the camera can be recovered from object-level embeddings using linear and non-linear probes. Across a range of detection transformer models, our results show a surprisingly strong and previously unknown ability of 2D DETR models to represent useful information about the 3D properties of objects, despite the complete lack of 3D supervision during model pre-training.