Agent Identity Evals: Measuring Agentic Identity
This work addresses the challenge of ensuring stable identity in LMAs for developers and users to improve agentic capabilities and trust, though it appears incremental as it builds on existing evaluation methods without claiming broad SOTA gains.
The paper tackles the problem of language model agents (LMAs) lacking stable identity due to inherited pathologies from large language models, which undermines their reliability and trustworthiness, by introducing agent identity evals (AIE), a framework with novel metrics to measure and maintain agentic identity over time, including capabilities and recovery from perturbations.
Central to agentic capability and trustworthiness of language model agents (LMAs) is the extent they maintain stable, reliable, identity over time. However, LMAs inherit pathologies from large language models (LLMs) (statelessness, stochasticity, sensitivity to prompts and linguistically-intermediation) which can undermine their identifiability, continuity, persistence and consistency. This attrition of identity can erode their reliability, trustworthiness and utility by interfering with their agentic capabilities such as reasoning, planning and action. To address these challenges, we introduce \textit{agent identity evals} (AIE), a rigorous, statistically-driven, empirical framework for measuring the degree to which an LMA system exhibit and maintain their agentic identity over time, including their capabilities, properties and ability to recover from state perturbations. AIE comprises a set of novel metrics which can integrate with other measures of performance, capability and agentic robustness to assist in the design of optimal LMA infrastructure and scaffolding such as memory and tools. We set out formal definitions and methods that can be applied at each stage of the LMA life-cycle, and worked examples of how to apply them.