Towards Physically Safe Reinforcement Learning under Supervision
This work addresses safety concerns in reinforcement learning for robotics, though it is incremental in nature.
The paper tackles the problem of safely training a new robot control policy using a supervisor policy to accelerate learning while ensuring physical safety, achieving effective results in OpenAI Gym experiments.
This paper addresses the question of how a previously available control policy $π_s$ can be used as a supervisor to more quickly and safely train a new learned control policy $π_L$ for a robot. A weighted average of the supervisor and learned policies is used during trials, with a heavier weight initially on the supervisor, in order to allow safe and useful physical trials while the learned policy is still ineffective. During the process, the weight is adjusted to favor the learned policy. As weights are adjusted, the learned network must compensate so as to give safe and reasonable outputs under the different weights. A pioneer network is introduced that pre-learns a policy that performs similarly to the current learned policy under the planned next step for new weights; this pioneer network then replaces the currently learned network in the next set of trials. Experiments in OpenAI Gym demonstrate the effectiveness of the proposed method.