Fine-Tuning Vision-Language Models for Visual Navigation Assistance
This addresses indoor navigation accessibility for visually impaired people, but it is incremental as it builds on existing vision-language models.
The paper tackled indoor navigation for visually impaired individuals by fine-tuning the BLIP-2 model with LoRA to generate step-by-step instructions, resulting in significant improvements in directional instruction generation.
We address vision-language-driven indoor navigation to assist visually impaired individuals in reaching a target location using images and natural language guidance. Traditional navigation systems are ineffective indoors due to the lack of precise location data. Our approach integrates vision and language models to generate step-by-step navigational instructions, enhancing accessibility and independence. We fine-tune the BLIP-2 model with Low Rank Adaptation (LoRA) on a manually annotated indoor navigation dataset. We propose an evaluation metric that refines the BERT F1 score by emphasizing directional and sequential variables, providing a more comprehensive measure of navigational performance. After applying LoRA, the model significantly improved in generating directional instructions, overcoming limitations in the original BLIP-2 model.