Reinforcement Learning for Control of Non-Markovian Cellular Population Dynamics
This addresses the challenge of drug dosing for adaptive cell populations like cancer, with clinical relevance, though it is incremental as it applies existing RL methods to a new domain.
The paper tackled the problem of controlling cell populations with non-Markovian dynamics using reinforcement learning, and found that model-free deep RL recovered exact solutions and achieved control even with long-range temporal dynamics, noise, and varying memory strength.
Many organisms and cell types, from bacteria to cancer cells, exhibit a remarkable ability to adapt to fluctuating environments. Additionally, cells can leverage a memory of past environments to better survive previously-encountered stressors. From a control perspective, this adaptability poses significant challenges in driving cell populations toward extinction, and thus poses an open question with great clinical significance. In this work, we focus on drug dosing in cell populations exhibiting phenotypic plasticity. For specific dynamical models switching between resistant and susceptible states, exact solutions are known. However, when the underlying system parameters are unknown, and for complex memory-based systems, obtaining the optimal solution is currently intractable. To address this challenge, we apply reinforcement learning (RL) to identify informed dosing strategies to control cell populations evolving under novel non-Markovian dynamics. We find that model-free deep RL is able to recover exact solutions and control cell populations even in the presence of long-range temporal dynamics. To further test our approach in more realistic settings, we demonstrate robust RL-based control strategies in environments with measurement noise and dynamic memory strength.