VORTEX: Aligning Task Utility and Human Preferences through LLM-Guided Reward Shaping
This work addresses the challenge of integrating evolving human preferences into optimization systems for stakeholders in social impact domains, offering a novel paradigm but with incremental elements.
The paper tackles the problem of aligning AI decision systems with human preferences expressed in natural language while preserving core utility guarantees, proposing VORTEX, a framework that uses LLMs for reward shaping and achieves Pareto-optimal trade-offs in real-world allocation tasks.
In social impact optimization, AI decision systems often rely on solvers that optimize well-calibrated mathematical objectives. However, these solvers cannot directly accommodate evolving human preferences, typically expressed in natural language rather than formal constraints. Recent approaches address this by using large language models (LLMs) to generate new reward functions from preference descriptions. While flexible, they risk sacrificing the system's core utility guarantees. In this paper, we propose \texttt{VORTEX}, a language-guided reward shaping framework that preserves established optimization goals while adaptively incorporating human feedback. By formalizing the problem as multi-objective optimization, we use LLMs to iteratively generate shaping rewards based on verbal reinforcement and text-gradient prompt updates. This allows stakeholders to steer decision behavior via natural language without modifying solvers or specifying trade-off weights. We provide theoretical guarantees that \texttt{VORTEX} converges to Pareto-optimal trade-offs between utility and preference satisfaction. Empirical results in real-world allocation tasks demonstrate that \texttt{VORTEX} outperforms baselines in satisfying human-aligned coverage goals while maintaining high task performance. This work introduces a practical and theoretically grounded paradigm for human-AI collaborative optimization guided by natural language.