Towards Multimodal Multitask Scene Understanding Models for Indoor Mobile Agents
This addresses the challenge of insufficient labeled data and multimodal fusion for indoor mobile agents, though it is incremental as it builds on existing multitask and multimodal approaches.
The paper tackles the problem of indoor scene understanding for mobile agents by proposing a multimodal multitask model that handles RGB images and Lidar inputs to perform tasks like 3D object detection, depth completion, human pose estimation, and semantic segmentation, achieving improvements such as an 11.7% boost in 3D object detection on the ARKitScenes dataset.
The perception system in personalized mobile agents requires developing indoor scene understanding models, which can understand 3D geometries, capture objectiveness, analyze human behaviors, etc. Nonetheless, this direction has not been well-explored in comparison with models for outdoor environments (e.g., the autonomous driving system that includes pedestrian prediction, car detection, traffic sign recognition, etc.). In this paper, we first discuss the main challenge: insufficient, or even no, labeled data for real-world indoor environments, and other challenges such as fusion between heterogeneous sources of information (e.g., RGB images and Lidar point clouds), modeling relationships between a diverse set of outputs (e.g., 3D object locations, depth estimation, and human poses), and computational efficiency. Then, we describe MMISM (Multi-modality input Multi-task output Indoor Scene understanding Model) to tackle the above challenges. MMISM considers RGB images as well as sparse Lidar points as inputs and 3D object detection, depth completion, human pose estimation, and semantic segmentation as output tasks. We show that MMISM performs on par or even better than single-task models; e.g., we improve the baseline 3D object detection results by 11.7% on the benchmark ARKitScenes dataset.