OctoSense: Self-Supervised Learning for Multimodal Robot Perception
For robotics researchers needing robust multimodal perception, this work provides a new dataset and a fast, self-supervised architecture that outperforms existing methods, though the approach is incremental.
OctoSense introduces an open-source multimodal sensor platform and dataset with 59 hours of driving data, and proposes a late-fusion masked autoencoder for self-supervised learning that achieves 6.68 ms inference on NVIDIA 5090, outperforms image-only foundation models on optical flow, depth, segmentation, and ego-motion, and maintains robustness under degraded sensory conditions.
We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.