On the validity of kernel approximations for orthogonally-initialized neural networks
arXiv:2104.05878v13 citations
Originality Synthesis-oriented
AI Analysis
This work provides a theoretical extension for orthogonal initialization in neural networks, but it is incremental as it builds on existing kernel approximation frameworks.
The authors extended kernel function approximation results from neural networks with Gaussian weights to those initialized with Haar-distributed random orthogonal matrices, using random matrix theory.
In this note we extend kernel function approximation results for neural networks with Gaussian-distributed weights to single-layer networks initialized using Haar-distributed random orthogonal matrices (with possible rescaling). This is accomplished using recent results from random matrix theory.