Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
This work addresses the problem of improving value function accuracy for reinforcement learning practitioners, though it appears incremental as it builds on existing decomposition ideas.
The paper tackles the challenge of value estimation in reinforcement learning by decomposing the value function into dynamics and return components, resulting in a deep RL algorithm that achieves superior performance in MuJoCo continuous control tasks, including under delayed reward settings.
Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the quality of reward signals. In this paper, we propose a two-step understanding of value estimation from the perspective of future prediction, through decomposing the value function into a reward-independent future dynamics part and a policy-independent trajectory return part. We then derive a practical deep RL algorithm from the above decomposition, consisting of a convolutional trajectory representation model, a conditional variational dynamics model to predict the expected representation of future trajectory and a convex trajectory return model that maps a trajectory representation to its return. Our algorithm is evaluated in MuJoCo continuous control tasks and shows superior results under both common settings and delayed reward settings.