Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration
This addresses the challenge of robust human-robot collaboration in open-ended scenarios, though it is incremental as it builds on prior work by combining language and action cues.
The paper tackles the problem of robots inferring ambiguous or novel human goals in collaborative tasks by integrating natural language preferences with observed actions, resulting in more stable goal predictions and significantly fewer mistakes compared to baselines.
To collaborate with humans, robots must infer goals that are often ambiguous, difficult to articulate, or not drawn from a fixed set. Prior approaches restrict inference to a predefined goal set, rely only on observed actions, or depend exclusively on explicit instructions, making them brittle in real-world interactions. We present BALI (Bidirectional Action-Language Inference) for goal prediction, a method that integrates natural language preferences with observed human actions in a receding-horizon planning tree. BALI combines language and action cues from the human, asks clarifying questions only when the expected information gain from the answer outweighs the cost of interruption, and selects supportive actions that align with inferred goals. We evaluate the approach in collaborative cooking tasks, where goals may be novel to the robot and unbounded. Compared to baselines, BALI yields more stable goal predictions and significantly fewer mistakes.