Dual Optimal: Make Your LLM Peer-like with Dignity
This addresses the issue of sycophancy and evasiveness in AI assistants for users, representing a novel method rather than an incremental improvement.
The paper tackles the problem of aligned language models exhibiting 'Evasive Servant' behavior by proposing the Dignified Peer framework, which uses the PersonaKnob dataset and a constrained Lagrangian DPO algorithm to build an LLM agent with dignity and peer-like qualities, as validated through psychometrically calibrated evaluation.
Current aligned language models exhibit a dual failure mode we term the Evasive Servant: they sycophantically validate flawed user beliefs while deflecting responsibility with boilerplate disclaimers. We propose the Dignified Peer framework, which counters servility with anti-sycophancy and trustworthiness, and mitigates evasiveness through empathy and creativity. Realizing this agent requires overcoming significant challenges in data supervision, objective collapse, and evaluation bias. We address these issues by introducing the PersonaKnob dataset which features a compositional partial order structure of multiple persona preference. This data is utilized alongside a tolerant constrained Lagrangian DPO algorithm that dynamically balances all persona dimensions to prevent behavioral collapse. Additionally, we employ a psychometrically calibrated Item Response Theory evaluation protocol to disentangle latent model persona capability from confounders like judge biases. Extensive empirical studies demonstrate that our approach successfully build a LLM agent with both dignity and peer.