MetricOpt: Learning to Optimize Black-Box Evaluation Metrics
This addresses the challenge of optimizing arbitrary metrics for machine learning practitioners, offering a pluggable solution that avoids complex loss design, though it is incremental as it builds on existing optimizers.
The paper tackles the problem of directly optimizing non-differentiable evaluation metrics like misclassification rate and recall in a black-box setting, achieving state-of-the-art performance on tasks such as image classification, retrieval, and object detection with consistent improvements over competing methods.
We study the problem of directly optimizing arbitrary non-differentiable task evaluation metrics such as misclassification rate and recall. Our method, named MetricOpt, operates in a black-box setting where the computational details of the target metric are unknown. We achieve this by learning a differentiable value function, which maps compact task-specific model parameters to metric observations. The learned value function is easily pluggable into existing optimizers like SGD and Adam, and is effective for rapidly finetuning a pre-trained model. This leads to consistent improvements since the value function provides effective metric supervision during finetuning, and helps to correct the potential bias of loss-only supervision. MetricOpt achieves state-of-the-art performance on a variety of metrics for (image) classification, image retrieval and object detection. Solid benefits are found over competing methods, which often involve complex loss design or adaptation. MetricOpt also generalizes well to new tasks and model architectures.