Polyharmonic Cascade
This provides a novel deep learning approach for researchers in machine learning, though it appears incremental as it builds on existing spline and cascade concepts.
The paper tackles the problem of approximating complex nonlinear functions while maintaining global smoothness and a probabilistic interpretation, by introducing the polyharmonic cascade architecture and a training method that solves a global linear system per batch, achieving fast learning without overfitting on MNIST.
This paper presents a deep machine learning architecture, the "polyharmonic cascade" -- a sequence of packages of polyharmonic splines, where each layer is rigorously derived from the theory of random functions and the principles of indifference. This makes it possible to approximate nonlinear functions of arbitrary complexity while preserving global smoothness and a probabilistic interpretation. For the polyharmonic cascade, a training method alternative to gradient descent is proposed: instead of directly optimizing the coefficients, one solves a single global linear system on each batch with respect to the function values at fixed "constellations" of nodes. This yields synchronized updates of all layers, preserves the probabilistic interpretation of individual layers and theoretical consistency with the original model, and scales well: all computations reduce to 2D matrix operations efficiently executed on a GPU. Fast learning without overfitting on MNIST is demonstrated.