CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
This work addresses the challenge of enabling robots to find arbitrary objects based on language descriptions without in-domain training, which is incremental as it builds on existing open-vocabulary models.
The paper tackles the problem of language-driven zero-shot object navigation (L-ZSON) for robots, introducing the Pasture benchmark and a framework called CLIP on Wheels (CoW) that adapts open-vocabulary models without fine-tuning. The result shows that a simple CoW baseline matches the navigation efficiency of a state-of-the-art method trained on 500M steps and provides a 15.6 percentage point improvement in success over another state-of-the-art model.
For robots to be generally useful, they must be able to find arbitrary objects described by people (i.e., be language-driven) even without expensive navigation training on in-domain data (i.e., perform zero-shot inference). We explore these capabilities in a unified setting: language-driven zero-shot object navigation (L-ZSON). Inspired by the recent success of open-vocabulary models for image classification, we investigate a straightforward framework, CLIP on Wheels (CoW), to adapt open-vocabulary models to this task without fine-tuning. To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects. We conduct an in-depth empirical study by directly deploying 21 CoW baselines across Habitat, RoboTHOR, and Pasture. In total, we evaluate over 90k navigation episodes and find that (1) CoW baselines often struggle to leverage language descriptions, but are proficient at finding uncommon objects. (2) A simple CoW, with CLIP-based object localization and classical exploration -- and no additional training -- matches the navigation efficiency of a state-of-the-art ZSON method trained for 500M steps on Habitat MP3D data. This same CoW provides a 15.6 percentage point improvement in success over a state-of-the-art RoboTHOR ZSON model.