On Feature Scaling of Recursive Feature Machines
This incremental work explores a specific behavior in RFMs, potentially connecting them to neural network phenomena for researchers in kernel methods and machine learning theory.
The authors investigated how Recursive Feature Machines (RFMs) behave when random noise features are added to regression datasets, finding that test Mean Squared Error (MSE) shows a decrease-increase-decrease pattern similar to the double descent phenomenon in neural networks.
In this technical report, we explore the behavior of Recursive Feature Machines (RFMs), a type of novel kernel machine that recursively learns features via the average gradient outer product, through a series of experiments on regression datasets. When successively adding random noise features to a dataset, we observe intriguing patterns in the Mean Squared Error (MSE) curves with the test MSE exhibiting a decrease-increase-decrease pattern. This behavior is consistent across different dataset sizes, noise parameters, and target functions. Interestingly, the observed MSE curves show similarities to the "double descent" phenomenon observed in deep neural networks, hinting at new connection between RFMs and neural network behavior. This report lays the groundwork for future research into this peculiar behavior.