A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernel
This work addresses a theoretical gap in machine learning by analyzing a special case of NTK extrapolation, which is incremental as it builds on existing linear extrapolation findings.
The paper tackles the problem of understanding extrapolation behavior in ReLU MLPs under the Neural Tangent Kernel (NTK) regime, specifically near the origin, and discovers that quadratic extrapolation occurs for evaluation points close to the origin, contrasting with known linear extrapolation far from the origin.
It has been demonstrated both theoretically and empirically that the ReLU MLP tends to extrapolate linearly for an out-of-distribution evaluation point. The machine learning literature provides ample analysis with respect to the mechanisms to which linearity is induced. However, the analysis of extrapolation at the origin under the NTK regime remains a more unexplored special case. In particular, the infinite-dimensional feature map induced by the neural tangent kernel is not translationally invariant. This means that the study of an out-of-distribution evaluation point very far from the origin is not equivalent to the evaluation of a point very near the origin. And since the feature map is rotation invariant, these two special cases may represent the most canonically extreme bounds of ReLU NTK extrapolation. Ultimately, it is this loose recognition of the two special cases of extrapolation that motivate the discovery of quadratic extrapolation for an evaluation close to the origin.