SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing
For developers deploying LLM agents, Safety Sentry reduces harmful actions while preserving user autonomy by replacing binary guard models with context-aware, adjustable routing.
LLM agents risk irreversible harm from misjudged tool calls. Safety Sentry reframes binary safe/unsafe labeling as a three-way routing decision (EXECUTE, ASK, REFUSE) per instance, using a lightweight guard model with a single decoding-time threshold for adjustable risk tolerance, outperforming baselines on accuracy and safety recall.
LLM agents act on real-world environments through tool calls, and a single misjudged action can cause irreversible harm. The standard safeguard is a guard model that labels each proposed action as safe or unsafe, but this binary view conflates two distinct decisions: whether the action is harmful in itself, and whether it is appropriate given the user's context. It also operates at the granularity of action categories rather than individual instances, producing routine interruptions that erode autonomy and train users to wave through the most consequential alerts. We reframe the problem as a per-instance three-way routing decision over {EXECUTE, ASK, REFUSE} and instantiate it with Safety Sentry, a lightweight guard model whose inference reduces to a single decoding call. A single decoding-time threshold lets one fixed checkpoint be re-positioned across deployments of differing risk tolerance without retraining. Safety Sentry outperforms a broad set of open-weight and frontier closed-source baselines on overall accuracy and safety-related recall, while controlling both directional error rates simultaneously.