ROOAD: RELLIS Off-road Odometry Analysis Dataset
This provides a new benchmark for researchers working on localization in off-road environments, though it is incremental as it focuses on dataset creation and evaluation.
The paper introduces the ROOAD dataset, a high-quality off-road visual-inertial dataset, and evaluates it on two VIO algorithms, finding they perform 2 to 30 times worse compared to structured environments, with OpenVINS showing better stability and VINS-Fusion better accuracy in some cases.
The development and implementation of visual-inertial odometry (VIO) has focused on structured environments, but interest in localization in off-road environments is growing. In this paper, we present the RELLIS Off-road Odometry Analysis Dataset (ROOAD) which provides high-quality, time-synchronized off-road monocular visual-inertial data sequences to further the development of related research. We evaluated the dataset on two state-of-the-art VIO algorithms, (1) Open-VINS and (2) VINS-Fusion. Our findings indicate that both algorithms perform 2 to 30 times worse on the ROOAD dataset compared to their performance in structured environments. Furthermore, OpenVINS has better tracking stability and real-time performance than VINS-Fusion in the off-road environment, while VINS-Fusion outperformed OpenVINS in tracking accuracy in several data sequences. Since the camera-IMU calibration tool from Kalibr toolkit is used extensively in this work, we have included several calibration data sequences. Our hand measurements show Kalibr's tool achieved +/-1 degree for orientation error and +/-1 mm at best (x- and y-axis) and +/-10 mm (z-axis) at worse for position error in the camera frame between the camera and IMU. This novel dataset provides a new set of scenarios for researchers to design and test their localization algorithms on, as well as critical insights in the current performance of VIO in off-road environments. ROOAD Dataset: github.com/unmannedlab/ROOAD