Beyond Static Evaluation: Rethinking the Assessment of Personalized Agent Adaptability in Information Retrieval
This work addresses the challenge of assessing agent adaptability for users in dynamic information retrieval contexts, but it is incremental as it builds on existing practices like LLM-driven simulation.
The paper tackles the problem of evaluating personalized AI agents in information retrieval by proposing a conceptual framework that shifts from static benchmarks to interaction-aware assessments, focusing on adaptability over time, with a case study in e-commerce search using the PersonalWAB dataset.
Personalized AI agents are becoming central to modern information retrieval, yet most evaluation methodologies remain static, relying on fixed benchmarks and one-off metrics that fail to reflect how users' needs evolve over time. These limitations hinder our ability to assess whether agents can meaningfully adapt to individuals across dynamic, longitudinal interactions. In this perspective paper, we propose a conceptual lens for rethinking evaluation in adaptive personalization, shifting the focus from static performance snapshots to interaction-aware, evolving assessments. We organize this lens around three core components: (1) persona-based user simulation with temporally evolving preference models; (2) structured elicitation protocols inspired by reference interviews to extract preferences in context; and (3) adaptation-aware evaluation mechanisms that measure how agent behavior improves across sessions and tasks. While recent works have embraced LLM-driven user simulation, we situate this practice within a broader paradigm for evaluating agents over time. To illustrate our ideas, we conduct a case study in e-commerce search using the PersonalWAB dataset. Beyond presenting a framework, our work lays a conceptual foundation for understanding and evaluating personalization as a continuous, user-centric endeavor.