Derivative of the truncated singular value and eigen decomposition
This work addresses a technical bottleneck for researchers and practitioners using automatic differentiation in fields like machine learning and computational physics, but it is incremental as it builds on previous work.
The paper tackles the problem of computing stable and efficient gradients for truncated singular value and eigenvalue decompositions, which are needed for automatic differentiation in machine learning and computational physics, by providing a detailed derivation of the derivative in terms of the truncated part without requiring full decomposition knowledge.
Recently developed applications in the field of machine learning and computational physics rely on automatic differentiation techniques, that require stable and efficient linear algebra gradient computations. This technical note provides a comprehensive and detailed discussion of the derivative of the truncated singular and eigenvalue decomposition. It summarizes previous work and builds on them with an extensive description of how to derive the relevant terms. A main focus is correctly expressing the derivative in terms of the truncated part, despite lacking knowledge of the full decomposition.