A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning
This work addresses foundational issues in model-based RL for researchers, highlighting limitations in widely used methods.
The paper tackles the discrepancy between model-based reinforcement learning's theoretical advantages and its poor empirical performance due to error compounding, and demonstrates that the MuZero loss fails in stochastic environments and has exponential sample complexity in deterministic ones.
This note clarifies some confusions (and perhaps throws out more) around model-based reinforcement learning and their theoretical understanding in the context of deep RL. Main topics of discussion are (1) how to reconcile model-based RL's bad empirical reputation on error compounding with its superior theoretical properties, and (2) the limitations of empirically popular losses. For the latter, concrete counterexamples for the "MuZero loss" are constructed to show that it not only fails in stochastic environments, but also suffers exponential sample complexity in deterministic environments when data provides sufficient coverage.