Multi-Agent Robotic Control with Onboard Vision-Language Models
For roboticists seeking deployable, low-latency autonomous systems, this work provides a practical MAS architecture that eliminates cloud dependency, though it is an incremental application of existing VLM and MAS techniques to a specific domain.
This paper presents a multi-agent system (MAS) architecture using onboard vision-language models (3-20B parameters) for controlling a mobile manipulator in a simulated warehouse, achieving viable performance across five task categories without external compute. The system was validated in hardware-in-the-loop simulation, demonstrating cost-efficient alternative to cloud-dependent deployments.
Vision Language Models (VLMs) and Vision Language Action (VLA) models have shown promise in robotic control. Yet, they face significant challenges regarding explainability, generalization, and compute requirements. This paper presents a Multi-Agent System (MAS) architecture that addresses these limitations by deploying specialized agents on onboard hardware - eliminating dependence on external compute. The system controls a multi-purpose autonomous mobile manipulator in a simulated industrial warehouse, fulfilling five task categories: safety inspection, warehouse maintenance, warehouse search, package quality verification, and responding to human requests. Compact VLMs (3-20B parameters) are used throughout, with fine-tuning applied to improve package inspection accuracy. A novel "Megamind" orchestration agent mitigates context retention issues inherent to long-horizon planning with smaller models. The system was validated in a hardware-in-the-loop simulation using an AMD Ryzen(TM) AI mini PC. Results demonstrate that a fully onboard MAS architecture is a viable, cost-efficient alternative to cloud-dependent deployments, with strong potential for real-world transfer. The simulation environment has been released as open source under the Apache 2.0 licence.