Precise Asymptotic Analysis of Deep Random Feature Models
This provides theoretical insights for researchers in machine learning theory, though it is incremental as it builds on existing random feature model analyses.
The authors tackled the problem of understanding the performance of deep random feature models in regression by deriving exact asymptotic expressions, showing that depth affects performance even when only the last layer is trained.
We provide exact asymptotic expressions for the performance of regression by an $L-$layer deep random feature (RF) model, where the input is mapped through multiple random embedding and non-linear activation functions. For this purpose, we establish two key steps: First, we prove a novel universality result for RF models and deterministic data, by which we demonstrate that a deep random feature model is equivalent to a deep linear Gaussian model that matches it in the first and second moments, at each layer. Second, we make use of the convex Gaussian Min-Max theorem multiple times to obtain the exact behavior of deep RF models. We further characterize the variation of the eigendistribution in different layers of the equivalent Gaussian model, demonstrating that depth has a tangible effect on model performance despite the fact that only the last layer of the model is being trained.