Multi-Objective Optimization for Self-Adjusting Weighted Gradient in Machine Learning Tasks
This addresses the problem of suboptimal loss function design for machine learning practitioners, offering an incremental improvement over standard methods.
The paper tackled the limitation of using a linear combination of loss functions in machine learning by proposing a multi-objective optimization approach using the hypervolume indicator, which achieved better mean loss than direct optimization on a denoising autoencoder task.
Much of the focus in machine learning research is placed in creating new architectures and optimization methods, but the overall loss function is seldom questioned. This paper interprets machine learning from a multi-objective optimization perspective, showing the limitations of the default linear combination of loss functions over a data set and introducing the hypervolume indicator as an alternative. It is shown that the gradient of the hypervolume is defined by a self-adjusting weighted mean of the individual loss gradients, making it similar to the gradient of a weighted mean loss but without requiring the weights to be defined a priori. This enables an inner boosting-like behavior, where the current model is used to automatically place higher weights on samples with higher losses but without requiring the use of multiple models. Results on a denoising autoencoder show that the new formulation is able to achieve better mean loss than the direct optimization of the mean loss, providing evidence to the conjecture that self-adjusting the weights creates a smoother loss surface.