HERO'S JOURNEY: Testing Complex Rule Induction with Text Games
For researchers evaluating LLMs' complex reasoning, this benchmark reveals limitations in rule induction, especially procedural tasks, highlighting an open challenge.
HERO'S JOURNEY is a benchmark for rule induction in goal-directed episodic tasks. State-of-the-art LLMs show limited and uneven rule induction ability, with process execution being a bottleneck and procedural induction remaining an open challenge.
We introduce HERO'S JOURNEY, a benchmark for rule induction in goal-directed episodic tasks, where agents must infer hidden rules from demonstrations and act on them through multi-step execution. HERO'S JOURNEY covers eight tasks across attribute and procedural induction families, each with four structural rule forms, controllable lexical grounding, and identifiability conditions. Evaluating state-of-the-art LLMs, we find that models show evidence of rule induction, but the ability is limited and uneven across tasks. Meanwhile, process execution adds an execution bottleneck for models, whereas surface semantics has minimal effect. Induction-specific steering methods improve performance on attribute tasks but show no reliable gains on procedural tasks, suggesting the gap in procedural induction remains an open challenge.