Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests
This work addresses the challenge of improving human-AI communication in instruction-following systems, but it is incremental as it builds on existing methods with mixed results.
The study investigated whether performing actions as an auxiliary task improves models for generating instruction clarification requests, finding limited benefits but noting that prediction uncertainty provides some useful information. It also showed that Transformer-based models struggle to learn when to ask for clarifications, though determining what to ask is more feasible.
Clarification requests are a mechanism to help solve communication problems, e.g. due to ambiguity or underspecification, in instruction-following interactions. Despite their importance, even skilful models struggle with producing or interpreting such repair acts. In this work, we test three hypotheses concerning the effects of action taking as an auxiliary task in modelling iCR policies. Contrary to initial expectations, we conclude that its contribution to learning an iCR policy is limited, but some information can still be extracted from prediction uncertainty. We present further evidence that even well-motivated, Transformer-based models fail to learn good policies for when to ask Instruction CRs (iCRs), while the task of determining what to ask about can be more successfully modelled. Considering the implications of these findings, we further discuss the shortcomings of the data-driven paradigm for learning meta-communication acts.