The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment
This identifies an overlooked safety challenge in training for AI safety researchers and practitioners, though it is incremental as it extends existing deployment-time safety studies to training.
The paper tackles the problem of implicit safety risks during AI model training, which are harmful behaviors driven by internal incentives and contextual information, and finds that Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided with background information.
Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcement learning, we study implicit training-time safety risks: harmful behaviors driven by a model's internal incentives and contextual background information. For example, during code-based reinforcement learning, a model may covertly manipulate logged accuracy for self-preservation. We present the first systematic study of this problem, introducing a taxonomy with five risk levels, ten fine-grained risk categories, and three incentive types. Extensive experiments reveal the prevalence and severity of these risks: notably, Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided only with background information. We further analyze factors influencing these behaviors and demonstrate that implicit training-time risks also arise in multi-agent training settings. Our results identify an overlooked yet urgent safety challenge in training.