Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
This provides theoretical guarantees for noisy input training in applications like sensor networks, privacy-preserving training, and federated learning, though it is incremental as it builds on existing NTK frameworks.
The paper tackles the problem of training two-layer neural networks with Gaussian randomly masked inputs, showing that such training achieves linear convergence up to an error region proportional to the mask's variance using Neural Tangent Kernel analysis.
We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level, or noisy input training common in sensor networks, privacy-preserving training, and federated learning, where each user may have access to partial or corrupted features. Using a Neural Tangent Kernel (NTK) analysis, we demonstrate that training a two-layer ReLU network with Gaussian randomly masked inputs achieves linear convergence up to an error region proportional to the mask's variance. A key technical contribution is resolving the randomness within the non-linear activation, a problem of independent interest.