QoNext: Towards Next-generation QoE for Foundation Models
This addresses the need for better user-centric evaluation in foundation models, though it is incremental by applying existing QoE principles to a new domain.
The authors tackled the problem of evaluating foundation models by moving beyond output correctness to capture user experience during interaction, introducing QoNext as a framework that adapts Quality of Experience principles to assess foundation models through human studies and predictive models.
Existing evaluations of foundation models, including recent human-centric approaches, fail to capture what truly matters: user's experience during interaction. Current methods treat evaluation as a matter of output correctness alone, overlooking that user satisfaction emerges from the interplay between response quality and interaction, which limits their ability to account for the mechanisms underlying user experience. To address this gap, we introduce QoNext, the first framework that adapts Quality of Experience (QoE) principles from networking and multimedia to the assessment of foundation models. QoNext identifies experiential factors that shape user experience and incorporates them into controlled experiments, where human ratings are collected under varied configurations. From these studies we construct a QoE-oriented database and train predictive models that estimate perceived user experience from measurable system parameters. Our results demonstrate that QoNext not only enables proactive and fine-grained evaluation but also provides actionable guidance for productized services of optimizing foundation models in practice.