Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions
This work addresses the challenge of enabling more proactive and adaptable AI agents in dynamic environments, representing a novel paradigm rather than an incremental improvement.
The paper tackles the problem of reinforcement learning agents being limited to passive adaptation in fixed environments by introducing model-changing actions that allow agents to actively modify world dynamics, resulting in a framework for jointly optimizing configuration and primitive action policies to improve long-term rewards.
Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.