Inference in Multi-Layer Networks with Matrix-Valued Unknowns
This work addresses inference challenges in deep generative models and neural network learning, but it is incremental as it extends an existing algorithm to matrix-valued cases.
The authors tackled the problem of inferring input and hidden variables in stochastic multi-layer neural networks with matrix-valued unknowns, proposing a unified algorithm (ML-Mat-VAMP) that extends prior work to handle matrices, and showed its performance can be exactly predicted in a large-system limit, enabling precise predictions for parameter and test error in learning tasks.
We consider the problem of inferring the input and hidden variables of a stochastic multi-layer neural network from an observation of the output. The hidden variables in each layer are represented as matrices. This problem applies to signal recovery via deep generative prior models, multi-task and mixed regression and learning certain classes of two-layer neural networks. A unified approximation algorithm for both MAP and MMSE inference is proposed by extending a recently-developed Multi-Layer Vector Approximate Message Passing (ML-VAMP) algorithm to handle matrix-valued unknowns. It is shown that the performance of the proposed Multi-Layer Matrix VAMP (ML-Mat-VAMP) algorithm can be exactly predicted in a certain random large-system limit, where the dimensions $N\times d$ of the unknown quantities grow as $N\rightarrow\infty$ with $d$ fixed. In the two-layer neural-network learning problem, this scaling corresponds to the case where the number of input features and training samples grow to infinity but the number of hidden nodes stays fixed. The analysis enables a precise prediction of the parameter and test error of the learning.