Modulation of viability signals for self-regulatory control
This work addresses the challenge of enabling more autonomous and adaptive AI agents by shifting from reward-driven to preference-driven learning, which could benefit fields like robotics and autonomous systems, though it appears incremental as it builds on existing active inference frameworks.
The paper tackles the problem of how agents can learn to adapt their behavior by self-supervising their preferences, rather than relying on external rewards, in reinforcement learning tasks. It demonstrates this approach in a dynamic environment, comparing model-free and model-based agents that minimize surprisal and expected free energy, respectively.
We revisit the role of instrumental value as a driver of adaptive behavior. In active inference, instrumental or extrinsic value is quantified by the information-theoretic surprisal of a set of observations measuring the extent to which those observations conform to prior beliefs or preferences. That is, an agent is expected to seek the type of evidence that is consistent with its own model of the world. For reinforcement learning tasks, the distribution of preferences replaces the notion of reward. We explore a scenario in which the agent learns this distribution in a self-supervised manner. In particular, we highlight the distinction between observations induced by the environment and those pertaining more directly to the continuity of an agent in time. We evaluate our methodology in a dynamic environment with discrete time and actions. First with a surprisal minimizing model-free agent (in the RL sense) and then expanding to the model-based case to minimize the expected free energy.