Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms
This work addresses initialization challenges in tensor networks for researchers in machine learning and quantum physics, but it is incremental as it builds on existing tensor network methods.
The paper tackles the problem of initializing tensorized neural networks and tensor network algorithms by developing two algorithms that use partial Frobenius and lineal entrywise norms to normalize subnetworks, preventing divergence or zero norms. The result includes scaling analysis for MPS/TT and MPO/TT-M layers, with code made publicly available.
We present two algorithms to initialize layers of tensorized neural networks and general tensor network algorithms using partial computations of their Frobenius norms and lineal entrywise norms, depending on the type of tensor network involved. The core of this method is the use of the norm of subnetworks of the tensor network in an iterative way, so that we normalize by the finite values of the norms that led to the divergence or zero norm. In addition, the method benefits from the reuse of intermediate calculations. We have also applied it to the Matrix Product State/Tensor Train (MPS/TT) and Matrix Product Operator/Tensor Train Matrix (MPO/TT-M) layers and have seen its scaling versus the number of nodes, bond dimension, and physical dimension. All code is publicly available.