Dual Advantage Fields
For offline goal-conditioned reinforcement learning, DAF provides a principled method to convert global value estimates into local action preferences, outperforming baselines in tasks requiring non-direct movement.
Dual Advantage Fields (DAF) extracts policies from bilinear dual value models by using goal embeddings as gradients to compute local advantage signals, improving performance on OGBench locomotion, manipulation, and puzzle tasks.
Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state. We propose Dual Advantage Fields, a policy-extraction method that turns a bilinear dual value model into a local advantage signal. Under bilinear dual parameterization, the goal embedding is the gradient of the value field with respect to the state representation. DAF learns an action-effect model that predicts the discounted feature displacement induced by an action and scores actions by the alignment between this displacement and the goal direction. In the realizable case, this score equals the goal-conditioned Bellman advantage, yielding a standard local policy-improvement guarantee. On OGBench locomotion, manipulation, and puzzle tasks, DAF improves aggregate RLiable metrics and performs strongly in settings where locally correct actions differ from direct movement toward the final goal.