All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation
This addresses the challenge of flexible long-term deployment for VLN agents across multiple environments, though it appears incremental as it builds on parameter-efficient adapters like LoRA.
The paper tackles the problem of catastrophic forgetting in vision-and-language navigation agents when adapting to diverse scenes, formalizing it as the all-day multi-scenes lifelong VLN problem, and proposes Tucker Adaptation (TuKA) with a decoupled knowledge incremental learning strategy, resulting in AlldayWalker outperforming state-of-the-art baselines in experiments.
Deploying vision-and-language navigation (VLN) agents requires adaptation across diverse scenes and environments, but fine-tuning on a specific scenario often causes catastrophic forgetting in others, which severely limits flexible long-term deployment. We formalize this challenge as the all-day multi-scenes lifelong VLN (AML-VLN) problem. Existing parameter-efficient adapters (e.g., LoRA and its variants) are limited by their two-dimensional matrix form, which fails to capture the multi-hierarchical navigation knowledge spanning multiple scenes and environments. To address this, we propose Tucker Adaptation (TuKA), which represents the multi-hierarchical navigation knowledge as a high-order tensor and leverages Tucker decomposition to decouple the knowledge into shared subspaces and scenario-specific experts. We further introduce a decoupled knowledge incremental learning strategy to consolidate shared subspaces while constraining specific experts for decoupled lifelong learning. Building on TuKA, we also develop a VLN agent named AlldayWalker, which continually learns across multiple navigation scenarios, achieving all-day multi-scenes navigation. Extensive experiments show that AlldayWalker consistently outperforms state-of-the-art baselines.