Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning
This work addresses the challenge of optimizing loss functions for machine learning models, which is an incremental advancement in the field of meta-learning and loss function learning.
The paper tackles the problem of improving model performance by learning loss functions through a meta-learning framework, achieving results where the discovered loss functions outperform cross-entropy and state-of-the-art methods across various neural network architectures and datasets.
In this paper, we develop upon the emerging topic of loss function learning, which aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for learning model-agnostic loss functions via a hybrid neuro-symbolic search approach. The framework first uses evolution-based methods to search the space of primitive mathematical operations to find a set of symbolic loss functions. Second, the set of learned loss functions are subsequently parameterized and optimized via an end-to-end gradient-based training procedure. The versatility of the proposed framework is empirically validated on a diverse set of supervised learning tasks. Results show that the meta-learned loss functions discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods on a diverse range of neural network architectures and datasets.