RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization
This work addresses safety-critical domains where online data collection is infeasible, offering a solution that balances risk and performance, though it is incremental as it builds on prior risk-averse and risk-neutral methods.
The paper tackles the problem of achieving both high returns and low catastrophic risk in offline reinforcement learning by introducing the RAMAC framework, which combines an expressive generative actor with a distributional critic to improve risk-sensitive performance, resulting in consistent gains in CVaR while maintaining strong returns on most Stochastic-D4RL tasks.
In safety-critical domains where online data collection is infeasible, offline reinforcement learning (RL) offers an attractive alternative but only if policies deliver high returns without incurring catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of value conservatism and restricted policy classes, whereas expressive policies are only used in risk-neutral settings. Here, we address this gap by introducing the \textbf{Risk-Aware Multimodal Actor-Critic (RAMAC)} framework, which couples an \emph{expressive generative actor} with a distributional critic. The RAMAC differentiates composite objective combining distributional risk and BC loss through the generative path, achieving risk-sensitive learning in complex multimodal scenarios. We instantiate RAMAC with diffusion and flow-matching actors and observe consistent gains in $\mathrm{CVaR}_{0.1}$ while maintaining strong returns on most Stochastic-D4RL tasks. Code: https://github.com/KaiFukazawa/RAMAC.git