Visual Language Models as Operator Agents in the Space Domain
This work addresses autonomous decision-making for space operations, but it is incremental as it builds on existing multimodal models.
The paper tackles the problem of autonomous control in space missions by applying Vision-Language Models (VLMs) as operator agents, showing they can effectively process visual and textual data to perform orbital maneuvers in simulation and inspect physical objects, competing with traditional methods.
This paper explores the application of Vision-Language Models (VLMs) as operator agents in the space domain, focusing on both software and hardware operational paradigms. Building on advances in Large Language Models (LLMs) and their multimodal extensions, we investigate how VLMs can enhance autonomous control and decision-making in space missions. In the software context, we employ VLMs within the Kerbal Space Program Differential Games (KSPDG) simulation environment, enabling the agent to interpret visual screenshots of the graphical user interface to perform complex orbital maneuvers. In the hardware context, we integrate VLMs with robotic systems equipped with cameras to inspect and diagnose physical space objects, such as satellites. Our results demonstrate that VLMs can effectively process visual and textual data to generate contextually appropriate actions, competing with traditional methods and non-multimodal LLMs in simulation tasks, and showing promise in real-world applications.