MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception
This work addresses the problem of occlusions in vision systems for robotics and computer vision applications, providing a new tool for non-line-of-sight perception.
The authors tackled the problem of non-line-of-sight perception by creating a millimeter-wave dataset and simulator, MITO, which captured over 24 million mmWave frames and generated 550 high-resolution images. The dataset and simulator enable broader NLOS perception, with applications in vision systems.
The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment. However, optical perception can often be disrupted due to common occurrences, such as occlusions, which can pose challenges to existing vision systems. We present MITO, the first millimeter-wave (mmWave) dataset of diverse, everyday objects, collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Unlike visible light, mmWave signals can penetrate common occlusions (e.g., cardboard boxes, fabric, plastic) but each mmWave frame has much lower resolution than typical cameras. To capture higher-resolution mmWave images, we leverage the robot's mobility and fuse frames over the synthesized aperture. MITO captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave (synthetic aperture) images in line-of-sight and non-light-of-sight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals, taken from 76 different objects. We develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh. Finally, we demonstrate the utility of our dataset and simulator for enabling broader NLOS perception by developing benchmarks for NLOS segmentation and classification.