Fast and Efficient Local Search for Genetic Programming Based Loss Function Learning
This addresses the problem of optimizing loss functions for better model training across various domains, though it builds incrementally on existing loss function learning approaches.
The paper tackles the problem of learning loss functions that improve model performance by proposing a hybrid meta-learning framework combining genetic programming and unrolled differentiation. Results show the learned loss functions improve convergence, sample efficiency, and inference performance across diverse supervised learning tasks.
In this paper, we develop upon the topic of loss function learning, an emergent meta-learning paradigm that aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach. The framework first uses genetic programming to find a set of symbolic loss functions. Second, the set of learned loss functions is subsequently parameterized and optimized via unrolled differentiation. The versatility and performance of the proposed framework are empirically validated on a diverse set of supervised learning tasks. Results show that the learned loss functions bring improved convergence, sample efficiency, and inference performance on tabulated, computer vision, and natural language processing problems, using a variety of task-specific neural network architectures.