Retrospective Learning from Interactions
This addresses the challenge of enhancing LLM performance in interactive settings for users by leveraging natural feedback signals, though it is incremental as it builds on existing interaction paradigms.
The paper tackles the problem of improving large language models (LLMs) by learning from implicit feedback in multi-turn interactions, achieving a task completion rate increase from 31% to 82% in a multimodal reasoning scenario without external annotations.
Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. We introduce ReSpect, a method to learn from such signals in past interactions via retrospection without additional annotations. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct a multimodal LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.