Inverse++: Vision-Centric 3D Semantic Occupancy Prediction Assisted with 3D Object Detection
This work addresses the critical problem of detecting small dynamic objects like pedestrians and bicycles for autonomous driving safety, though it is incremental as it builds on existing methods by adding an auxiliary task.
The paper tackles 3D semantic occupancy prediction for autonomous vehicles by introducing a multitask learning approach with a 3D object detection auxiliary branch, achieving state-of-the-art results with an IoU score of 31.73% and a mIoU score of 20.91% on the nuScenes dataset, including improved detection of vulnerable road users.
3D semantic occupancy prediction aims to forecast detailed geometric and semantic information of the surrounding environment for autonomous vehicles (AVs) using onboard surround-view cameras. Existing methods primarily focus on intricate inner structure module designs to improve model performance, such as efficient feature sampling and aggregation processes or intermediate feature representation formats. In this paper, we explore multitask learning by introducing an additional 3D supervision signal by incorporating an additional 3D object detection auxiliary branch. This extra 3D supervision signal enhances the model's overall performance by strengthening the capability of the intermediate features to capture small dynamic objects in the scene, and these small dynamic objects often include vulnerable road users, i.e. bicycles, motorcycles, and pedestrians, whose detection is crucial for ensuring driving safety in autonomous vehicles. Extensive experiments conducted on the nuScenes datasets, including challenging rainy and nighttime scenarios, showcase that our approach attains state-of-the-art results, achieving an IoU score of 31.73% and a mIoU score of 20.91% and excels at detecting vulnerable road users (VRU). The code will be made available at:https://github.com/DanielMing123/Inverse++