Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation
This work addresses the need for more detailed diagnostics in VLN to improve language-guided navigation systems, though it is incremental as it builds on existing evaluation methods.
This study tackled the problem of evaluating Vision-Language Navigation (VLN) models by developing a fine-grained framework based on context-free grammar and LLMs to generate data across five instruction categories, revealing performance discrepancies such as stagnation in numerical comprehension and biases in directional concepts.
This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finer-grained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e. direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems.