Reinforcement Learning Under Algorithmic Triage
This addresses the challenge of integrating reinforcement learning with human-in-the-loop systems, which is incremental as it extends algorithmic triage from supervised to reinforcement learning settings.
The paper tackles the problem of developing reinforcement learning models optimized for algorithmic triage, where machines and humans share decision-making, by proposing a two-stage actor-critic method. The result shows that in a synthetic car driving task, their method effectively complements human policies and outperforms several baselines.
Methods to learn under algorithmic triage have predominantly focused on supervised learning settings where each decision, or prediction, is independent of each other. Under algorithmic triage, a supervised learning model predicts a fraction of the instances and humans predict the remaining ones. In this work, we take a first step towards developing reinforcement learning models that are optimized to operate under algorithmic triage. To this end, we look at the problem through the framework of options and develop a two-stage actor-critic method to learn reinforcement learning models under triage. The first stage performs offline, off-policy training using human data gathered in an environment where the human has operated on their own. The second stage performs on-policy training to account for the impact that switching may have on the human policy, which may be difficult to anticipate from the above human data. Extensive simulation experiments in a synthetic car driving task show that the machine models and the triage policies trained using our two-stage method effectively complement human policies and outperform those provided by several competitive baselines.