Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics
This provides a general and efficient solution method for macroeconomic models, addressing nontrivial market-clearing conditions that traditional methods struggle with, though it is incremental in applying reinforcement learning to known bottlenecks in macroeconomics.
The paper tackles the challenge of solving heterogeneous agent models with aggregate risk by introducing a structural reinforcement learning method that replaces cross-sectional distributions with low-dimensional prices as state variables, enabling agents to learn equilibrium price dynamics from simulations; this approach efficiently solves models like Krusell-Smith, Huggett, and HANK globally within minutes.
We present a new approach to formulating and solving heterogeneous agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let agents learn equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting agents' structural knowledge of their own individual dynamics. Our SRL method yields a general and highly efficient global solution method for heterogeneous agent models that sidesteps the Master equation and handles problems traditional methods struggle with, in particular nontrivial market-clearing conditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which we solve globally within minutes.