Integral Quadratic Constraints for Repeated ReLU
This work addresses stability analysis for recurrent neural networks in control synthesis, offering incremental improvements over prior IQC-based methods.
The paper tackles the problem of analyzing stability and performance of recurrent neural networks with ReLU activations by introducing new dynamic integral quadratic constraints (IQCs) for repeated ReLU, which provide less conservative bounds than existing methods, as demonstrated in a numerical example.
This paper presents a new dynamic integral quadratic constraint (IQC) for the repeated Rectified Linear Unit (ReLU). These dynamic IQCs can be used to analyze stability and induced $\ell_2$-gain performance of discrete-time, recurrent neural networks (RNNs) with ReLU activation functions. These analysis conditions can be incorporated into learning-based controller synthesis methods, which currently rely on static IQCs. We show that our proposed dynamic IQCs for repeated ReLU form a superset of the dynamic IQCs for repeated, slope-restricted nonlinearities. We also prove that the $\ell_2$-gain bounds are nonincreasing with respect to the horizon used in the dynamic IQC filter. A numerical example using a simple (academic) RNN shows that our proposed IQCs lead to less conservative bounds than existing IQCs.