TraCeS: Trajectory Based Credit Assignment From Sparse Safety Feedback
This addresses the challenge of specifying safety constraints in RL for applications where safety is critical but hard to define, offering a scalable solution for continuous control.
The paper tackles the problem of learning unknown safety definitions in reinforcement learning from sparse binary labels, proposing a trajectory-based credit assignment method to estimate step-level safety impacts and reformulating safe RL to optimize policies, with empirical results showing effectiveness in satisfying unknown safety constraints across continuous control tasks.
In safe reinforcement learning (RL), auxiliary safety costs are used to align the agent to safe decision making. In practice, safety constraints, including cost functions and budgets, are unknown or hard to specify, as it requires anticipation of all possible unsafe behaviors. We therefore address a general setting where the true safety definition is unknown, and has to be learned from sparsely labeled data. Our key contributions are: first, we design a safety model that performs credit assignment to estimate each decision step's impact on the overall safety using a dataset of diverse trajectories and their corresponding binary safety labels (i.e., whether the corresponding trajectory is safe/unsafe). Second, we illustrate the architecture of our safety model to demonstrate its ability to learn a separate safety score for each timestep. Third, we reformulate the safe RL problem using the proposed safety model and derive an effective algorithm to optimize a safe yet rewarding policy. Finally, our empirical results corroborate our findings and show that this approach is effective in satisfying unknown safety definition, and scalable to various continuous control tasks.