Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision
This work addresses the need for interpretable and modular robot policies in safety-critical applications, offering an incremental improvement by bridging high-dimensional learning with symbolic control.
The paper tackled the problem of opaque end-to-end visuomotor policies in robotics by proposing a neuro-symbolic approach where a vision-language model synthesizes interpretable Behavior Tree policies from multimodal inputs, and experiments on robotic manipulators showed successful transfer to physical systems with results indicating adaptability for structured decision-making.
Vision-language models (VLMs) have recently demonstrated strong capabilities in mapping multimodal observations to robot behaviors. However, most current approaches rely on end-to-end visuomotor policies that remain opaque and difficult to analyze, limiting their use in safety-critical robotic applications. In contrast, classical robotic systems often rely on structured policy representations that provide interpretability, modularity, and reactive execution. This work investigates how foundation models can be specialized to generate structured robot policies grounded in multimodal perception, bridging high-dimensional learning and symbolic control. We propose a neuro-symbolic approach in which a VLM synthesizes executable Behavior Tree policies from visual observations, natural language instructions, and structured system specifications. To enable scalable supervision without manual annotation, we introduce an automated pipeline that generates a synthetic multimodal dataset of domain-randomized scenes paired with instruction-policy examples produced by a foundation model. Real-world experiments on two robotic manipulators show that structured policies learned entirely from synthetic supervision transfer successfully to physical systems. The results indicate that foundation models can be adapted to produce interpretable and structured robot policies, providing an alternative to opaque end-to-end approaches for multimodal robot decision making.