Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors
This addresses the problem of reward hacking in reinforcement learning for game-like environments, but it is incremental as it empirically evaluates existing methods without proposing a new solution.
The study investigated how intrinsic motivation (IM) techniques affect reinforcement learning agent behaviors in reward-sparse games, finding that IM increases initial rewards but alters gameplay, while Generalized Reward Matching (GRM) partially mitigates reward hacking.
Games are challenging for Reinforcement Learning~(RL) agents due to their reward-sparsity, as rewards are only obtainable after long sequences of deliberate actions. Intrinsic Motivation~(IM) methods -- which introduce exploration rewards -- are an effective solution to reward-sparsity. However, IM also causes an issue known as `reward hacking' where the agent optimizes for the new reward at the expense of properly playing the game. The larger problem is that reward hacking itself is largely unknown; there is no answer to whether, and to what extent, IM rewards change the behavior of RL agents. This study takes a first step by empirically evaluating the impact on behavior of three IM techniques on the MiniGrid game-like environment. We compare these IM models with Generalized Reward Matching~(GRM), a method that can be used with any intrinsic reward function to guarantee optimality. Our results suggest that IM causes noticeable change by increasing the initial rewards, but also altering the way the agent plays; and that GRM mitigated reward hacking in some scenarios.