Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
This work addresses challenges in reinforcement learning for agents needing to learn efficiently from imperfect data, representing a novel method for a known bottleneck.
The paper tackles the problem of learning optimal policies in reinforcement learning by introducing a novel value function Q(s,s') that estimates the utility of transitioning between states, enabling off-policy learning from sub-optimal or random state observations. The result is a method that decouples actions from values, offering benefits such as value function transfer and improved learning in redundant action spaces.
In this paper, we introduce a novel form of value function, $Q(s, s')$, that expresses the utility of transitioning from a state $s$ to a neighboring state $s'$ and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at http://sites.google.com/view/qss-paper.