Learning Pareto manifolds in high dimensions: How can regularization help?
This addresses the problem of improving generalization in multi-objective machine learning for applications like fairness and distribution shifts, though it is incremental as it builds on existing regularization concepts.
The paper tackles the challenge of multi-objective learning in high-dimensional settings by proposing a two-stage framework that leverages low-dimensional structure, demonstrating effectiveness in multi-distribution learning and fairness-risk trade-offs.
Simultaneously addressing multiple objectives is becoming increasingly important in modern machine learning. At the same time, data is often high-dimensional and costly to label. For a single objective such as prediction risk, conventional regularization techniques are known to improve generalization when the data exhibits low-dimensional structure like sparsity. However, it is largely unexplored how to leverage this structure in the context of multi-objective learning (MOL) with multiple competing objectives. In this work, we discuss how the application of vanilla regularization approaches can fail, and propose a two-stage MOL framework that can successfully leverage low-dimensional structure. We demonstrate its effectiveness experimentally for multi-distribution learning and fairness-risk trade-offs.