Comparative Design Space Exploration of Dense and Semi-Dense SLAM
This provides a quantitative evaluation framework for SLAM researchers and developers, though it is incremental as it builds on existing tools.
The paper tackled the lack of holistic benchmarking for SLAM systems by extending SLAMBench to compare KinectFusion and LSD-SLAM on accuracy, energy consumption, and frame rate across desktop and embedded hardware, analyzing individual kernels and design spaces.
SLAM has matured significantly over the past few years, and is beginning to appear in serious commercial products. While new SLAM systems are being proposed at every conference, evaluation is often restricted to qualitative visualizations or accuracy estimation against a ground truth. This is due to the lack of benchmarking methodologies which can holistically and quantitatively evaluate these systems. Further investigation at the level of individual kernels and parameter spaces of SLAM pipelines is non-existent, which is absolutely essential for systems research and integration. We extend the recently introduced SLAMBench framework to allow comparing two state-of-the-art SLAM pipelines, namely KinectFusion and LSD-SLAM, along the metrics of accuracy, energy consumption, and processing frame rate on two different hardware platforms, namely a desktop and an embedded device. We also analyze the pipelines at the level of individual kernels and explore their algorithmic and hardware design spaces for the first time, yielding valuable insights.