Active Learning MPC Objective Functions from Preferences
For control engineers and researchers, this work reduces the labeling effort required to learn MPC objective functions from human preferences, though it is an incremental improvement over existing active learning methods.
The paper addresses the challenge of learning the objective function for Model Predictive Control from human preferences, proposing two active learning strategies that reduce the number of human queries needed. The strategies achieve closed-loop behaviors more aligned with preferences using fewer queries than random sampling.
Designing the objective function in Model Predictive Control (MPC) is challenging when performance assessment criteria are available only from human judgment. We adopt a preference-based learning (PbL) approach to learn the MPC objective function from preferences over trajectory pairs. However, the real-world application of PbL is often restricted by the significant cost or limited availability of human preference queries. To address this, Active Learning (AL) strategies seek to improve sampling efficiency, reducing the labeling effort required to obtain a well-performing classifier. We present two AL strategies for learning the MPC objective function from human preferences over pairwise system trajectories: a pool-based strategy that selects trajectory pairs that are both uncertain under the current surrogate and diverse relative to previously labeled comparisons, and a query-synthesis strategy that incorporates new trajectories using the current surrogate-driven MPC. Numerical results show that the proposed strategies yield closed-loop behaviors that align more with the expressed preference using fewer number of queries compared to a random sampling approach.