TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matching
This addresses the issue of inefficient model learning in MBRL for robotic control, representing an incremental improvement over standard methods.
The paper tackles the problem of model-based reinforcement learning (MBRL) wasting model capacity on irrelevant data by proposing a transition occupancy matching (TOM) objective that focuses model learning on policy-relevant transitions. The result shows that TOM drives policies to higher task rewards faster than alternative approaches on Mujoco continuous robotic control tasks.
Standard model-based reinforcement learning (MBRL) approaches fit a transition model of the environment to all past experience, but this wastes model capacity on data that is irrelevant for policy improvement. We instead propose a new "transition occupancy matching" (TOM) objective for MBRL model learning: a model is good to the extent that the current policy experiences the same distribution of transitions inside the model as in the real environment. We derive TOM directly from a novel lower bound on the standard reinforcement learning objective. To optimize TOM, we show how to reduce it to a form of importance weighted maximum-likelihood estimation, where the automatically computed importance weights identify policy-relevant past experiences from a replay buffer, enabling stable optimization. TOM thus offers a plug-and-play model learning sub-routine that is compatible with any backbone MBRL algorithm. On various Mujoco continuous robotic control tasks, we show that TOM successfully focuses model learning on policy-relevant experience and drives policies faster to higher task rewards than alternative model learning approaches.