Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery
This work addresses the need for interpretable decision-making in healthcare, offering a method to bridge the gap between flexible machine learning and clinical transparency, though it is incremental in combining existing techniques for a specific bottleneck.
The paper tackles the problem of interpretability in reinforcement learning by proposing a nonparametric additive model for estimating value functions, with results validated through simulation and application to spine disease recovery, uncovering clinically aligned recommendations.
We propose a nonparametric additive model for estimating interpretable value functions in reinforcement learning, with an application in optimizing postoperative recovery through personalized, adaptive recommendations. While reinforcement learning has achieved significant success in various domains, recent methods often rely on black-box approaches such as neural networks, which hinder the examination of individual feature contributions to a decision-making policy. Our novel method offers a flexible technique for estimating action-value functions without explicit parametric assumptions, overcoming the limitations of the linearity assumption of classical algorithms. By incorporating local kernel regression and basis expansion, we obtain a sparse, additive representation of the action-value function, enabling local approximation and retrieval of nonlinear, independent contributions of select state features and the interactions between joint feature pairs. We validate our approach through a simulation study and apply it to spine disease recovery, uncovering recommendations aligned with clinical knowledge. This method bridges the gap between flexible machine learning techniques and the interpretability required in healthcare applications, paving the way for more personalized interventions.