NeuralSolver: Learning Algorithms For Consistent and Efficient Extrapolation Across General Tasks
This addresses the challenge of consistent and efficient extrapolation across general tasks for machine learning practitioners, representing a novel method for a known bottleneck.
The paper tackles the problem of learning algorithms from smaller problems to extrapolate efficiently to larger ones, introducing NeuralSolver, which outperforms prior state-of-the-art recurrent solvers in extrapolation tasks with smaller training problems and fewer parameters.
We contribute NeuralSolver, a novel recurrent solver that can efficiently and consistently extrapolate, i.e., learn algorithms from smaller problems (in terms of observation size) and execute those algorithms in large problems. Contrary to previous recurrent solvers, NeuralSolver can be naturally applied in both same-size problems, where the input and output sizes are the same, and in different-size problems, where the size of the input and output differ. To allow for this versatility, we design NeuralSolver with three main components: a recurrent module, that iteratively processes input information at different scales, a processing module, responsible for aggregating the previously processed information, and a curriculum-based training scheme, that improves the extrapolation performance of the method. To evaluate our method we introduce a set of novel different-size tasks and we show that NeuralSolver consistently outperforms the prior state-of-the-art recurrent solvers in extrapolating to larger problems, considering smaller training problems and requiring less parameters than other approaches.