Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards
This addresses the challenge of developing agents that can learn effectively without external rewards, which is crucial for advancing autonomous AI systems, though it appears incremental as it builds on existing intrinsic reward concepts.
The paper tackles the problem of achieving human-level performance in intrinsically-motivated play without extrinsic rewards by proposing a megalomania-driven intrinsic reward called mega-reward, which maximizes agents' control capabilities on entities, and it achieves performance comparable to Ex-PPO and professional human-level scores.
Intrinsic rewards were introduced to simulate how human intelligence works; they are usually evaluated by intrinsically-motivated play, i.e., playing games without extrinsic rewards but evaluated with extrinsic rewards. However, none of the existing intrinsic reward approaches can achieve human-level performance under this very challenging setting of intrinsically-motivated play. In this work, we propose a novel megalomania-driven intrinsic reward (called mega-reward), which, to our knowledge, is the first approach that achieves human-level performance in intrinsically-motivated play. Intuitively, mega-reward comes from the observation that infants' intelligence develops when they try to gain more control on entities in an environment; therefore, mega-reward aims to maximize the control capabilities of agents on given entities in a given environment. To formalize mega-reward, a relational transition model is proposed to bridge the gaps between direct and latent control. Experimental studies show that mega-reward (i) can greatly outperform all state-of-the-art intrinsic reward approaches, (ii) generally achieves the same level of performance as Ex-PPO and professional human-level scores, and (iii) has also a superior performance when it is incorporated with extrinsic rewards.