On the choice of the non-trainable internal weights in random feature maps
This work addresses a specific bottleneck in random feature maps for forecasting, offering an incremental improvement in efficiency and performance for practitioners in dynamical systems modeling.
The paper tackles the problem of selecting non-trainable internal weights in random feature maps to improve forecasting accuracy for dynamical systems, showing that a hit-and-run algorithm yields superior forecasting skill with orders of magnitude lower computational cost compared to learned neural networks.
The computationally cheap machine learning architecture of random feature maps can be viewed as a single-layer feedforward network in which the weights of the hidden layer are random but fixed and only the outer weights are learned via linear regression. The internal weights are typically chosen from a prescribed distribution. The choice of the internal weights significantly impacts the accuracy of random feature maps. We address here the task of how to best select the internal weights. In particular, we consider the forecasting problem whereby random feature maps are used to learn a one-step propagator map for a dynamical system. We provide a computationally cheap hit-and-run algorithm to select good internal weights which lead to good forecasting skill. We show that the number of good features is the main factor controlling the forecasting skill of random feature maps and acts as an effective feature dimension. Lastly, we compare random feature maps with single-layer feedforward neural networks in which the internal weights are now learned using gradient descent. We find that random feature maps have superior forecasting capabilities whilst having several orders of magnitude lower computational cost.