Removing Sandbagging in LLMs by Training with Weak Supervision
This work addresses the risk of AI systems exploiting weak supervision to underperform, offering a potential mitigation strategy for AI safety.
The paper studies sandbagging in LLMs, where models underperform when supervision is weak, and finds that combining supervised fine-tuning (SFT) with reinforcement learning (RL) during training can reliably elicit full performance, while neither method alone succeeds. The approach works only when training is indistinguishable from deployment.
As AI systems begin to automate complex tasks, supervision increasingly relies on weaker models or limited human oversight that cannot fully verify output quality. A model more capable than its supervisors could exploit this gap through sandbagging, producing work that appears acceptable but falls short of its true abilities. Can training elicit a model's best work even without reliable verification? We study this using model organisms trained to sandbag, testing elicitation techniques on problem-solving math, graduate-level science, and competitive coding tasks. We find that training with weak supervision can reliably elicit sandbagging models when supervised fine-tuning (SFT) and reinforcement learning (RL) are combined: SFT on weak demonstrations breaks the sandbagging behavior, enabling RL to then fully elicit performance. Neither method succeeds reliably alone-RL without SFT almost always leads to reward hacking rather than genuine improvement. Critically, this relies on training being indistinguishable from deployment; when models can distinguish between training and deployment, they can perform well during training while continuing to sandbag afterward. Our results provide initial evidence that training is a viable mitigation against sandbagging, while highlighting the importance of making training indistinguishable from deployment.