TrustAgent: Towards Safe and Trustworthy LLM-based Agents
This work addresses safety concerns for LLM-based agents in human-centric environments, representing an incremental improvement through a novel framework.
The paper tackles the problem of ensuring safety in LLM-based agents for high-stake domains by proposing TrustAgent, an Agent-Constitution-based framework that uses pre-planning, in-planning, and post-planning strategies to enhance safety and helpfulness, with experimental results showing effective identification and mitigation of potential dangers across multiple domains.
The rise of LLM-based agents shows great potential to revolutionize task planning, capturing significant attention. Given that these agents will be integrated into high-stake domains, ensuring their reliability and safety is crucial. This paper presents an Agent-Constitution-based agent framework, TrustAgent, with a particular focus on improving the LLM-based agent safety. The proposed framework ensures strict adherence to the Agent Constitution through three strategic components: pre-planning strategy which injects safety knowledge to the model before plan generation, in-planning strategy which enhances safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Our experimental results demonstrate that the proposed framework can effectively enhance an LLM agent's safety across multiple domains by identifying and mitigating potential dangers during the planning. Further analysis reveals that the framework not only improves safety but also enhances the helpfulness of the agent. Additionally, we highlight the importance of the LLM reasoning ability in adhering to the Constitution. This paper sheds light on how to ensure the safe integration of LLM-based agents into human-centric environments. Data and code are available at https://github.com/agiresearch/TrustAgent.