HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning
This work addresses the limitation of HRM in real-world scenarios where environments are dynamic and uncertain, though it appears incremental as it extends an existing model to new settings.
The paper tackles the problem of applying the Hierarchical Reasoning Model (HRM) to dynamic, uncertain, or partially observable environments by introducing HRM-Agent, a variant trained with reinforcement learning, and shows it can learn to navigate to goals in dynamic maze environments, with evidence of reusing computation from earlier time-steps.
The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM's strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the correct action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM's reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps.