HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer
This addresses a practical limitation in vehicle-to-vehicle collaboration where existing systems assume homogeneous sensors, potentially enabling more scalable and effective autonomous vehicle perception.
The paper tackles the problem of multi-agent hetero-modal cooperative perception where vehicles have different sensor types, presenting HM-ViT as the first unified framework for this task. It demonstrates superior performance over state-of-the-art methods on the V2V perception dataset OPV2V.
Vehicle-to-Vehicle technologies have enabled autonomous vehicles to share information to see through occlusions, greatly enhancing perception performance. Nevertheless, existing works all focused on homogeneous traffic where vehicles are equipped with the same type of sensors, which significantly hampers the scale of collaboration and benefit of cross-modality interactions. In this paper, we investigate the multi-agent hetero-modal cooperative perception problem where agents may have distinct sensor modalities. We present HM-ViT, the first unified multi-agent hetero-modal cooperative perception framework that can collaboratively predict 3D objects for highly dynamic vehicle-to-vehicle (V2V) collaborations with varying numbers and types of agents. To effectively fuse features from multi-view images and LiDAR point clouds, we design a novel heterogeneous 3D graph transformer to jointly reason inter-agent and intra-agent interactions. The extensive experiments on the V2V perception dataset OPV2V demonstrate that the HM-ViT outperforms SOTA cooperative perception methods for V2V hetero-modal cooperative perception. We will release codes to facilitate future research.