Safe Deep RL in 3D Environments using Human Feedback
This addresses the challenge of specifying safety constraints in real-world tasks where simulators are unavailable, though it is incremental as it builds on the ReQueST technique.
The paper tackled the problem of training safe deep reinforcement learning agents in complex 3D environments without a simulator, using human feedback to learn a neural simulator and reward model, resulting in an order of magnitude reduction in unsafe behavior compared to standard methods.
Agents should avoid unsafe behaviour during both training and deployment. This typically requires a simulator and a procedural specification of unsafe behaviour. Unfortunately, a simulator is not always available, and procedurally specifying constraints can be difficult or impossible for many real-world tasks. A recently introduced technique, ReQueST, aims to solve this problem by learning a neural simulator of the environment from safe human trajectories, then using the learned simulator to efficiently learn a reward model from human feedback. However, it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans - whether sufficient pixel-based neural simulator quality can be achieved, and whether the human data requirements are viable in terms of both quantity and quality. In this paper we answer this question in the affirmative, using ReQueST to train an agent to perform a 3D first-person object collection task using data entirely from human contractors. We show that the resulting agent exhibits an order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning.