Learning Effective Loss Functions Efficiently
This addresses the challenge of efficiently tuning loss functions for machine learning practitioners, offering a significant speedup over existing methods, though it is incremental as it builds on prior work in loss function optimization.
The paper tackles the problem of learning an optimal loss function to minimize validation error, presenting an algorithm that is asymptotically optimal in worst-case scenarios and efficient in easy cases, achieving orders-of-magnitude faster hyperparameter tuning than state-of-the-art methods and enabling on-the-fly learning of novel loss functions.
We consider the problem of learning a loss function which, when minimized over a training dataset, yields a model that approximately minimizes a validation error metric. Though learning an optimal loss function is NP-hard, we present an anytime algorithm that is asymptotically optimal in the worst case, and is provably efficient in an idealized "easy" case. Experimentally, we show that this algorithm can be used to tune loss function hyperparameters orders of magnitude faster than state-of-the-art alternatives. We also show that our algorithm can be used to learn novel and effective loss functions on-the-fly during training.