Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention
This work addresses the challenge of enabling autonomous vehicles to better understand and respond to passenger commands, representing an incremental improvement in multimodal fusion techniques for this domain.
The paper tackles the problem of language grounding for autonomous vehicles by localizing visual regions based on natural language commands, and introduces a layer fusion approach with region-specific dynamic layer attention to improve accuracy, outperforming state-of-the-art methods on the Talk2Car benchmark.
Grounding a command to the visual environment is an essential ingredient for interactions between autonomous vehicles and humans. In this work, we study the problem of language grounding for autonomous vehicles, which aims to localize a region in a visual scene according to a natural language command from a passenger. Prior work only employs the top layer representations of a vision-and-language pre-trained model to predict the region referred to by the command. However, such a method omits the useful features encoded in other layers, and thus results in inadequate understanding of the input scene and command. To tackle this limitation, we present the first layer fusion approach for this task. Since different visual regions may require distinct types of features to disambiguate them from each other, we further propose the region-specific dynamic (RSD) layer attention to adaptively fuse the multimodal information across layers for each region. Extensive experiments on the Talk2Car benchmark demonstrate that our approach helps predict more accurate regions and outperforms state-of-the-art methods.