Vision-Language Navigation with Energy-Based Policy
This work addresses a key bottleneck in vision-language navigation for AI agents by improving alignment with expert policies, though it is incremental as it builds on existing VLN architectures.
The paper tackles the problem of error accumulation and distribution mismatch in vision-language navigation by proposing an Energy-based Navigation Policy (ENP) that models joint state-action distributions, achieving promising performances on multiple benchmarks like R2R, REVERIE, RxR, and R2R-CE.
Vision-language navigation (VLN) requires an agent to execute actions following human instructions. Existing VLN models are optimized through expert demonstrations by supervised behavioural cloning or incorporating manual reward engineering. While straightforward, these efforts overlook the accumulation of errors in the Markov decision process, and struggle to match the distribution of the expert policy. Going beyond this, we propose an Energy-based Navigation Policy (ENP) to model the joint state-action distribution using an energy-based model. At each step, low energy values correspond to the state-action pairs that the expert is most likely to perform, and vice versa. Theoretically, the optimization objective is equivalent to minimizing the forward divergence between the occupancy measure of the expert and ours. Consequently, ENP learns to globally align with the expert policy by maximizing the likelihood of the actions and modeling the dynamics of the navigation states in a collaborative manner. With a variety of VLN architectures, ENP achieves promising performances on R2R, REVERIE, RxR, and R2R-CE, unleashing the power of existing VLN models.