COMPASS: Cognitive MCTS-Guided Process Alignment for Safe Search Agents
This work is significant for developers of LLM-powered search agents, as it provides a method to improve safety alignment against multi-step harmful interactions, which is an incremental improvement over existing alignment methods.
The paper addresses the problem of retrieval-induced safety degradation in LLM-powered search agents, where harmful intents can manifest through seemingly innocuous sub-queries. They propose COMPASS, a framework that uses cognitive tree exploration to synthesize attack trajectories and introspective step-wise alignment to identify risky intermediate actions, achieving a favorable safety-utility trade-off with less training data.
LLM-powered search agents enable multi-step reasoning and tool use. However, these capabilities introduce retrieval-induced safety degradation, as harmful intents may decompose into seemingly innocuous sub-queries that lead to unsafe outcomes. Existing alignment methods struggle to capture sparse safety signals and fail to supervise diverse violations across multi-step interactions. We propose COMPASS, a Cognitive MCTS-Guided Process Alignment framework designed to achieve robust safety alignment throughout the agent workflow while preserving general utility. COMPASS integrates cognitive tree exploration (CTE) to efficiently synthesize stealthy attack trajectories, and introspective step-wise alignment (ISA) to isolate risky intermediate actions for fine-grained process supervision. Empirical results show that COMPASS achieves a favorable safety-utility trade-off while requiring substantially less training data.