LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation
This addresses the problem of evaluating personalized AI assistants for researchers and developers by providing a more realistic benchmark, though it is incremental in improving evaluation methods.
The paper tackles the misalignment of existing benchmarks for personalized AI assistants by proposing LifeSim, a user simulator that models user cognition and generates coherent life trajectories, and LifeSim-Eval, a benchmark covering 8 domains and 1,200 scenarios. Experiments show that current LLMs have significant limitations in handling implicit intentions and long-term user preferences.
The rapid advancement of large language models (LLMs) has accelerated progress toward universal AI assistants. However, existing benchmarks for personalized assistants remain misaligned with real-world user-assistant interactions, failing to capture the complexity of external contexts and users' cognitive states. To bridge this gap, we propose LifeSim, a user simulator that models user cognition through the Belief-Desire-Intention (BDI) model within physical environments for coherent life trajectories generation, and simulates intention-driven user interactive behaviors. Based on LifeSim, we introduce LifeSim-Eval, a comprehensive benchmark for multi-scenario, long-horizon personalized assistance. LifeSim-Eval covers 8 life domains and 1,200 diverse scenarios, and adopts a multi-turn interactive method to assess models' abilities to complete explicit and implicit intentions, recover user profiles, and produce high-quality responses. Under both single-scenario and long-horizon settings, our experiments reveal that current LLMs face significant limitations in handling implicit intention and long-term user preference modeling.