Efficient RL for optimizing conversation level outcomes with an LLM-based tutor
This work addresses the challenge of aligning LLM-based tutors with long-term educational goals for students, though it is incremental as it builds on existing RLHF frameworks.
The paper tackles the problem of optimizing long-term outcomes in multi-turn dialogue settings like online math tutoring, where existing RLHF frameworks fall short by focusing on immediate turn-level preferences. The proposed method uses a lower-dimensional latent state representation of the student and a lightweight policy to improve long-term outcomes, showing improved results in LLM-simulated tutoring tasks.
Large language models (LLMs) built on existing reinforcement learning with human feedback (RLHF) frameworks typically optimize responses based on immediate turn-level human preferences. However, this approach falls short in multi-turn dialogue settings, such as online math tutoring. We propose a method to enhance LLM-based tutors by representing the dialogue history with a lower-dimensional latent state representation of a student and optimizing a long-term policy to determine high-level actions based on the latent state. The goal is to better align the tutor's behavior with the long-term objective of guiding the student towards solving a target math problem on their own. Our model is lightweight, requiring less computational resources than prior work of training the tutor policy end-to-end to directly output the tutor's next utterance. Our experiment results demonstrate that these modifications lead to improved long-term outcomes compared to prompting in LLM-simulated tutoring tasks.