Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning
This addresses safety-critical decision-making under uncertainty, such as in healthcare, but is incremental as it builds on prior dead-end discovery approaches.
The paper tackles the problem of identifying worst-case outcomes (dead-ends) in safety-critical offline reinforcement learning by proposing a framework that estimates distributions of expected returns to enable tunable risk-based detection. It demonstrates that DistDeD improves over prior methods by providing risk indications 10 hours earlier on average and increasing detection by 20% in a toy domain and ICU patient assessment.
In safety-critical decision-making scenarios being able to identify worst-case outcomes, or dead-ends is crucial in order to develop safe and reliable policies in practice. These situations are typically rife with uncertainty due to unknown or stochastic characteristics of the environment as well as limited offline training data. As a result, the value of a decision at any time point should be based on the distribution of its anticipated effects. We propose a framework to identify worst-case decision points, by explicitly estimating distributions of the expected return of a decision. These estimates enable earlier indication of dead-ends in a manner that is tunable based on the risk tolerance of the designed task. We demonstrate the utility of Distributional Dead-end Discovery (DistDeD) in a toy domain as well as when assessing the risk of severely ill patients in the intensive care unit reaching a point where death is unavoidable. We find that DistDeD significantly improves over prior discovery approaches, providing indications of the risk 10 hours earlier on average as well as increasing detection by 20%.