Approximating Discontinuous Nash Equilibrial Values of Two-Player General-Sum Differential Games
This work addresses safety issues in robotics applications where discontinuous rewards cause poor controller performance, offering an incremental improvement over existing self-supervised learning methods.
The paper tackled the problem of approximating discontinuous Nash equilibria in two-player differential games, which is crucial for safety in robotics, by proposing a hybrid method that combines supervised and self-supervised learning; results showed it achieved better safety performance than baseline methods in simulations with 5D and 9D state spaces.
Finding Nash equilibrial policies for two-player differential games requires solving Hamilton-Jacobi-Isaacs (HJI) PDEs. Self-supervised learning has been used to approximate solutions of such PDEs while circumventing the curse of dimensionality. However, this method fails to learn discontinuous PDE solutions due to its sampling nature, leading to poor safety performance of the resulting controllers in robotics applications when player rewards are discontinuous. This paper investigates two potential solutions to this problem: a hybrid method that leverages both supervised Nash equilibria and the HJI PDE, and a value-hardening method where a sequence of HJIs are solved with a gradually hardening reward. We compare these solutions using the resulting generalization and safety performance in two vehicle interaction simulation studies with 5D and 9D state spaces, respectively. Results show that with informative supervision (e.g., collision and near-collision demonstrations) and the low cost of self-supervised learning, the hybrid method achieves better safety performance than the supervised, self-supervised, and value hardening approaches on equal computational budget. Value hardening fails to generalize in the higher-dimensional case without informative supervision. Lastly, we show that the neural activation function needs to be continuously differentiable for learning PDEs and its choice can be case dependent.