Technical Report: Evaluating Goal Drift in Language Model Agents
This addresses the challenge of ensuring safe operation for autonomous language model agents, particularly in extended deployments without human oversight, and is incremental in proposing a novel analysis method.
The paper tackles the problem of goal drift in language model agents, where agents deviate from their original objectives over time, and finds that while the best-performing agent maintains nearly perfect adherence for over 100,000 tokens, all models exhibit some degree of drift.
As language models (LMs) are increasingly deployed as autonomous agents, their robust adherence to human-assigned objectives becomes crucial for safe operation. When these agents operate independently for extended periods without human oversight, even initially well-specified goals may gradually shift. Detecting and measuring goal drift - an agent's tendency to deviate from its original objective over time - presents significant challenges, as goals can shift gradually, causing only subtle behavioral changes. This paper proposes a novel approach to analyzing goal drift in LM agents. In our experiments, agents are first explicitly given a goal through their system prompt, then exposed to competing objectives through environmental pressures. We demonstrate that while the best-performing agent (a scaffolded version of Claude 3.5 Sonnet) maintains nearly perfect goal adherence for more than 100,000 tokens in our most difficult evaluation setting, all evaluated models exhibit some degree of goal drift. We also find that goal drift correlates with models' increasing susceptibility to pattern-matching behaviors as the context length grows.