Aligning Dialogue Agents with Global Feedback via Large Language Model Reward Decomposition
This addresses the challenge of reducing manual effort in reward shaping for dialogue agents, though it is incremental as it builds on existing LLM and reinforcement learning techniques.
The authors tackled the problem of aligning dialogue agents with only session-level feedback by proposing a large language model (LLM) based reward decomposition framework, which improved conversation quality in human evaluations compared to state-of-the-art methods.
We propose a large language model based reward decomposition framework for aligning dialogue agents using only a single session-level feedback signal. We leverage the reasoning capabilities of a frozen, pretrained large language model (LLM) to infer fine-grained local implicit rewards by decomposing global, session-level feedback. Our first text-only variant prompts the LLM to perform reward decomposition using only the dialogue transcript. The second multimodal variant incorporates additional behavioral cues, such as pitch, gaze, and facial affect, expressed as natural language descriptions. These inferred turn-level rewards are distilled into a lightweight reward model, which we utilize for RL-based fine-tuning for dialogue generation. We evaluate both text-only and multimodal variants against state-of-the-art reward decomposition methods and demonstrate notable improvements in human evaluations of conversation quality, suggesting that LLMs are strong reward decomposers that obviate the need for manual reward shaping and granular human feedback.