Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
This addresses the challenge of efficiently enforcing orthogonal constraints in neural networks, particularly for RNNs, offering a more robust optimization method with potential benefits for stability and performance in sequence modeling tasks.
The authors tackled the problem of first-order optimization with orthogonal and unitary constraints in neural networks by introducing a parametrization based on Lie group theory, which transforms it into an unconstrained problem. They applied this to RNNs, creating expRNN, and demonstrated faster, accurate, and more stable convergence in tasks testing RNNs.
We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.