Mixing Deep Learning and Multiple Criteria Optimization: An Application to Distributed Learning with Multiple Datasets
This work addresses bias reduction in machine learning for applications using multiple datasets, but it appears incremental as it combines existing deep learning and optimization methods.
The paper tackles the problem of supervised learning by formulating it as a multiple criteria optimization model to minimize a vector loss function in the Pareto sense, and extends this to distributed learning with multiple datasets to reduce bias, with numerical experiments conducted on MNIST digit classification.
The training phase is the most important stage during the machine learning process. In the case of labeled data and supervised learning, machine training consists in minimizing the loss function subject to different constraints. In an abstract setting, it can be formulated as a multiple criteria optimization model in which each criterion measures the distance between the output associated with a specific input and its label. Therefore, the fitting term is a vector function and its minimization is intended in the Pareto sense. We provide stability results of the efficient solutions with respect to perturbations of input and output data. We then extend the same approach to the case of learning with multiple datasets. The multiple dataset environment is relevant when reducing the bias due to the choice of a specific training set. We propose a scalarization approach to implement this model and numerical experiments in digit classification using MNIST data.