HorizonBench: Long-Horizon Personalization with Evolving Preferences
This work provides the first benchmark with ground-truth provenance for preference evolution over long horizons, enabling diagnosis of state-tracking failures in personalization systems.
HorizonBench introduces a benchmark for long-horizon personalization with evolving preferences, featuring 4,245 items from 360 simulated users with 6-month conversation histories. The best model achieves only 52.8% accuracy, with most models at or below 20% chance baseline, and over a third of errors stem from failing to update user state after preference changes.
User preferences evolve across months of interaction, and tracking them requires inferring when a stated preference has been changed by a subsequent life event. We define this problem as long-horizon personalization and observe that progress on it is limited by data availability and measurement, with no existing resource providing both naturalistic long-horizon interactions and the ground-truth provenance needed to diagnose why models fail. We introduce a data generator that produces conversations from a structured mental state graph, yielding ground-truth provenance for every preference change across 6-month timelines, and from it construct HorizonBench, a benchmark of 4,245 items from 360 simulated users with 6-month conversation histories averaging ~4,300 turns and ~163K tokens. HorizonBench provides a testbed for long-context modeling, memory-augmented architectures, theory-of-mind reasoning, and user modeling. Across 25 frontier models, the best model reaches 52.8% and most score at or below the 20% chance baseline. When these models err on evolved preferences, over a third of the time they select the user's originally stated value without tracking the updated user state. This belief-update failure persists across context lengths and expression explicitness levels, identifying state-tracking capability as the primary bottleneck for long-horizon personalization.