Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning
This work addresses the need for risk-aware foundations in continual RL, providing a novel theoretical framework for lifelong learning agents, though it is incremental as it builds on existing risk measure theory.
The paper tackles the problem of continual reinforcement learning (RL) by introducing ergodic risk measures, which extend risk-aware decision-making to the continual setting, showing their intuitive appeal and theoretical soundness through a case study and empirical results.
Continual reinforcement learning (continual RL) seeks to formalize the notions of lifelong learning and endless adaptation in RL. In particular, the aim of continual RL is to develop RL agents that can maintain a careful balance between retaining useful information and adapting to new situations. To date, continual RL has been explored almost exclusively through the lens of risk-neutral decision-making, in which the agent aims to optimize the expected (or mean) long-run performance. In this work, we present the first formal theoretical treatment of continual RL through the lens of risk-aware decision-making, in which the agent aims to optimize a reward-based measure of long-run performance beyond the mean. In particular, we show that the classical theory of risk measures, widely used as a theoretical foundation in non-continual risk-aware RL, is, in its current form, incompatible with the continual setting. Then, building on this insight, we extend risk measure theory into the continual setting by introducing a new class of ergodic risk measures that are compatible with continual learning. Finally, we provide a case study of risk-aware continual learning, along with empirical results, which show the intuitive appeal and theoretical soundness of ergodic risk measures.