Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed
This work addresses safety generalization for RL in healthcare, providing a benchmark for safety-critical domains, but it is incremental as it builds on existing safe RL methods with a new application.
The paper investigates whether safe reinforcement learning algorithms maintain safety guarantees under distribution shift, using diabetes management as a testbed, and finds that test-time shielding improves safety by achieving 13-14% Time-in-Range gains and reduces clinical risks across various algorithms and patient groups.
Safe Reinforcement Learning (RL) algorithms are typically evaluated under fixed training conditions. We investigate whether training-time safety guarantees transfer to deployment under distribution shift, using diabetes management as a safety-critical testbed. We benchmark safe RL algorithms on a unified clinical simulator and reveal a safety generalization gap: policies satisfying constraints during training frequently violate safety requirements on unseen patients. We demonstrate that test-time shielding, which filters unsafe actions using learned dynamics models, effectively restores safety across algorithms and patient populations. Across eight safe RL algorithms, three diabetes types, and three age groups, shielding achieves Time-in-Range gains of 13--14\% for strong baselines such as PPO-Lag and CPO while reducing clinical risk index and glucose variability. Our simulator and benchmark provide a platform for studying safety under distribution shift in safety-critical control domains. Code is available at https://github.com/safe-autonomy-lab/GlucoSim and https://github.com/safe-autonomy-lab/GlucoAlg.