Adaptive Teaching in Heterogeneous Agents: Balancing Surprise in Sparse Reward Scenarios
This addresses the challenge of adapting teaching methods for heterogeneous agents in sparse-reward scenarios, but it is incremental as it builds on existing LfD and surprise concepts.
The paper tackles the problem of inefficient learning in heterogeneous agents when a Student agent cannot replicate Teacher demonstrations due to capability discrepancies, by introducing a Teacher-Student framework that uses surprise to adapt demonstrations, resulting in improved Student learning in sparse-reward control tasks.
Learning from Demonstration (LfD) can be an efficient way to train systems with analogous agents by enabling ``Student'' agents to learn from the demonstrations of the most experienced ``Teacher'' agent, instead of training their policy in parallel. However, when there are discrepancies in agent capabilities, such as divergent actuator power or joint angle constraints, naively replicating demonstrations that are out of bounds for the Student's capability can limit efficient learning. We present a Teacher-Student learning framework specifically tailored to address the challenge of heterogeneity between the Teacher and Student agents. Our framework is based on the concept of ``surprise'', inspired by its application in exploration incentivization in sparse-reward environments. Surprise is repurposed to enable the Teacher to detect and adapt to differences between itself and the Student. By focusing on maximizing its surprise in response to the environment while concurrently minimizing the Student's surprise in response to the demonstrations, the Teacher agent can effectively tailor its demonstrations to the Student's specific capabilities and constraints. We validate our method by demonstrating improvements in the Student's learning in control tasks within sparse-reward environments.