OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions
This work addresses a critical problem for robotic perception by benchmarking semantic mapping under varying lighting, though it is incremental as it builds on existing OSM techniques.
The paper tackled the challenge of evaluating Open Semantic Mapping (OSM) algorithms under varying indoor lighting conditions by introducing OSMa-Bench, a benchmark with a novel dataset and automated pipeline, and found that it provides insights into model robustness through experiments on leading models like ConceptGraphs, BBQ, and OpenScene.
Open Semantic Mapping (OSM) is a key technology in robotic perception, combining semantic segmentation and SLAM techniques. This paper introduces a dynamically configurable and highly automated LLM/LVLM-powered pipeline for evaluating OSM solutions called OSMa-Bench (Open Semantic Mapping Benchmark). The study focuses on evaluating state-of-the-art semantic mapping algorithms under varying indoor lighting conditions, a critical challenge in indoor environments. We introduce a novel dataset with simulated RGB-D sequences and ground truth 3D reconstructions, facilitating the rigorous analysis of mapping performance across different lighting conditions. Through experiments on leading models such as ConceptGraphs, BBQ and OpenScene, we evaluate the semantic fidelity of object recognition and segmentation. Additionally, we introduce a Scene Graph evaluation method to analyze the ability of models to interpret semantic structure. The results provide insights into the robustness of these models, forming future research directions for developing resilient and adaptable robotic systems. Project page is available at https://be2rlab.github.io/OSMa-Bench/.