Priors are Powerful: Improving a Transformer for Multi-camera 3D Detection with 2D Priors
This is an incremental improvement for autonomous driving and robotics applications.
The paper tackles the problem of multi-camera 3D detection by using 2D priors from an image backbone to improve a transformer-based model, resulting in up to 12% higher average precision and faster convergence.
Transfomer-based approaches advance the recent development of multi-camera 3D detection both in academia and industry. In a vanilla transformer architecture, queries are randomly initialised and optimised for the whole dataset, without considering the differences among input frames. In this work, we propose to leverage the predictions from an image backbone, which is often highly optimised for 2D tasks, as priors to the transformer part of a 3D detection network. The method works by (1). augmenting image feature maps with 2D priors, (2). sampling query locations via ray-casting along 2D box centroids, as well as (3). initialising query features with object-level image features. Experimental results shows that 2D priors not only help the model converge faster, but also largely improve the baseline approach by up to 12% in terms of average precision.