Conservative Bias in Multi-Teacher Learning: Why Agents Prefer Low-Reward Advisors
This challenges fundamental assumptions about optimal teaching in RL and has implications for safety-critical robotic applications where human preferences align with this conservative bias.
This paper reveals that in interactive reinforcement learning, agents overwhelmingly prefer conservative, low-reward teachers over high-reward ones (93.16% selection rate), prioritizing consistency over optimality, and achieves 159% improvement over baseline Q-learning under concept drift.
Interactive reinforcement learning (IRL) has shown promise in enabling autonomous agents and robots to learn complex behaviours from human teachers, yet the dynamics of teacher selection remain poorly understood. This paper reveals an unexpected phenomenon in IRL: when given a choice between teachers with different reward structures, learning agents overwhelmingly prefer conservative, low-reward teachers (93.16% selection rate) over those offering 20x higher rewards. Through 1,250 experimental runs in navigation tasks with multiple expert teachers, we discovered: (1) Conservative bias dominates teacher selection: agents systematically choose the lowest-reward teacher, prioritising consistency over optimality; (2) Critical performance thresholds exist at teacher availability rho >= 0.6 and accuracy omega >= 0.6, below which the framework fails catastrophically; (3) The framework achieves 159% improvement over baseline Q-learning under concept drift. These findings challenge fundamental assumptions about optimal teaching in RL and suggest potential implications for human-robot collaboration, where human preferences for safety and consistency may align with the observed agent selection behaviour, potentially informing training paradigms for safety-critical robotic applications.