Lightweight Structured Multimodal Reasoning for Clinical Scene Understanding in Robotics
This work addresses the need for safe and interpretable robotic decision-making in dynamic clinical environments, such as surgery and patient monitoring, though it appears incremental by building on existing Vision-Language Models with structured enhancements.
The paper tackles the problem of robust multimodal perception and reasoning in healthcare robotics by developing a lightweight agentic framework that combines Qwen2.5-VL-3B-Instruct with SmolAgent-based orchestration for video-based scene understanding, achieving competitive accuracy and improved robustness on benchmarks like Video-MME and a custom clinical dataset.
Healthcare robotics requires robust multimodal perception and reasoning to ensure safety in dynamic clinical environments. Current Vision-Language Models (VLMs) demonstrate strong general-purpose capabilities but remain limited in temporal reasoning, uncertainty estimation, and structured outputs needed for robotic planning. We present a lightweight agentic multimodal framework for video-based scene understanding. Combining the Qwen2.5-VL-3B-Instruct model with a SmolAgent-based orchestration layer, it supports chain-of-thought reasoning, speech-vision fusion, and dynamic tool invocation. The framework generates structured scene graphs and leverages a hybrid retrieval module for interpretable and adaptive reasoning. Evaluations on the Video-MME benchmark and a custom clinical dataset show competitive accuracy and improved robustness compared to state-of-the-art VLMs, demonstrating its potential for applications in robot-assisted surgery, patient monitoring, and decision support.