OCTOPUS: Open-vocabulary Content Tracking and Object Placement Using Semantic Understanding in Mixed Reality
This addresses the challenge of automated object placement in AR for users, moving beyond closed-vocabulary limitations, though it appears incremental as it builds on existing models.
The paper tackles the problem of placing virtual content in natural locations in augmented reality by introducing an open-vocabulary method that uses a pipeline with segmentation models, vision-language models, and LLMs, achieving performance at least as good as human experts 57% of the time in a preliminary user study.
One key challenge in augmented reality is the placement of virtual content in natural locations. Existing automated techniques are only able to work with a closed-vocabulary, fixed set of objects. In this paper, we introduce a new open-vocabulary method for object placement. Our eight-stage pipeline leverages recent advances in segmentation models, vision-language models, and LLMs to place any virtual object in any AR camera frame or scene. In a preliminary user study, we show that our method performs at least as well as human experts 57% of the time.