Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization
For practitioners in deep learning who need to optimize a primary objective under secondary constraints (e.g., sparsity, compression), this provides a principled and effective method that respects objective hierarchy.
The paper addresses the problem of hierarchical multi-objective optimization where objectives have unequal importance, introducing Priority-Constrained Descent (PCD) that preserves primary descent while minimally distorting to guarantee progress on secondary objectives. PCD achieves Pareto dominance and better per-objective performance with interpretable trade-offs controlled by a single parameter τ.
Deep learning problems rarely involve objectives that are equal in importance. A primary objective defines the goal, whilst secondary objectives, such as sparsity, compression, or robustness constrain the solution. While existing multi-objective methods have proven effective in practice, they have a clear symmetry problem and neglect the inherent objective hierarchy built into these objective spaces. We introduce Priority-Constrained Descent (PCD), a gradient-based optimization framework designed to explicitly exploit hierarchical objective structures. PCD preserves the direction of primary descent whilst allowing for the minimal distortion necessary to guarantee progress on secondary objectives, controlled by a single $τ\in [0, 1]$ that dictates the strength of the distortion. The resulting formulation is invariant to objective scaling and admits exact closed-form solutions for problems with two and three objectives. We evaluate PCD within structured network compression settings, unstructured sparsity and low-rankness, and across a variety of synthetic experiments, showing Pareto dominance and better per-objective performance with secondary progress guarantees over existing methods, further exhibiting the interpretable trade-off that $τ$ provides.