One Reflection Suffice
This work addresses a practical bottleneck for researchers and practitioners using orthogonal weight matrices in deep learning, though it appears incremental as it builds on existing reflection-based methods.
The paper tackles the computational inefficiency of using many Householder reflections for orthogonal weight matrices in deep learning by proving that a single reflection, computed by an auxiliary neural network, is sufficient, thereby mitigating low GPU utilization.
Orthogonal weight matrices are used in many areas of deep learning. Much previous work attempt to alleviate the additional computational resources it requires to constrain weight matrices to be orthogonal. One popular approach utilizes *many* Householder reflections. The only practical drawback is that many reflections cause low GPU utilization. We mitigate this final drawback by proving that *one* reflection is sufficient, if the reflection is computed by an auxiliary neural network.