Safety Training Persists Through Helpfulness Optimization in LLM Agents
This work addresses the problem of maintaining safety in multi-step LLM agents for AI developers, though it is incremental as it builds on existing post-training methods.
The study investigated the persistence of safety training in LLM agents during sequential optimization of safety and helpfulness, finding that safety training remains robust through subsequent helpfulness training and that all configurations align near a linear Pareto frontier with R² = 0.77.
Safety post-training has been studied extensively in single-step "chat" settings where safety typically refers to refusing harmful requests. We study an "agentic" (i.e., multi-step, tool-use) setting where safety refers to harmful actions directly taken by the LLM. We compare the effects of running direct preference optimization (DPO) on safety or helpfulness alone vs both metrics sequentially. As expected, training on one metric alone results in an extreme point along this frontier. However, unlike prior work, we find that safety training persists through subsequent helpfulness training. We also find that all training configurations end up near a linear Pareto frontier with $R^2 = 0.77$. Even post-training on both metrics simultaneously simply results in another point on the frontier rather than finding a "best of both worlds" strategy, despite the presence of such strategies in our DPO dataset. Overall, our findings underscore the need for better understanding of post-training dynamics.