Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents
This work addresses safety constraints for LLM agents in robotics, which is crucial for practical deployments in collaborative environments, though it is incremental as it builds on existing LLM agent frameworks.
The paper tackles the problem of ensuring safety for LLM-driven robot agents by teaching them prohibited actions and enforcing compliance, proposing a safety constraint module based on linear temporal logic that enables natural language encoding, violation reasoning, and unsafe action pruning, with experiments in VirtualHome and on a real robot showing strict adherence to safety constraints and scalability.
Recent advancements in large language models (LLMs) have enabled a new research domain, LLM agents, for solving robotics and planning tasks by leveraging the world knowledge and general reasoning abilities of LLMs obtained during pretraining. However, while considerable effort has been made to teach the robot the "dos," the "don'ts" received relatively less attention. We argue that, for any practical usage, it is as crucial to teach the robot the "don'ts": conveying explicit instructions about prohibited actions, assessing the robot's comprehension of these restrictions, and, most importantly, ensuring compliance. Moreover, verifiable safe operation is essential for deployments that satisfy worldwide standards such as ISO 61508, which defines standards for safely deploying robots in industrial factory environments worldwide. Aiming at deploying the LLM agents in a collaborative environment, we propose a queryable safety constraint module based on linear temporal logic (LTL) that simultaneously enables natural language (NL) to temporal constraints encoding, safety violation reasoning and explaining, and unsafe action pruning. To demonstrate the effectiveness of our system, we conducted experiments in VirtualHome environment and on a real robot. The experimental results show that our system strictly adheres to the safety constraints and scales well with complex safety constraints, highlighting its potential for practical utility.