Which Factors Predict the Chat Experience of a Natural Language Generation Dialogue Service?
This work addresses improving user experience in dialogue systems for developers and users, but it is incremental as it builds on existing models without introducing a new paradigm.
The paper tackled predicting chat experience in natural language generation dialogue systems by proposing a conceptual model, achieving an R-square of 0.541 with 120 participants. It identified factors like user favorability, coherence, sentiment, and similarity as positive predictors, and suggested preferences for dialog agents with specific personality traits.
In this paper, we proposed a conceptual model to predict the chat experience in a natural language generation dialog system. We evaluated the model with 120 participants with Partial Least Squares Structural Equation Modeling (PLS-SEM) and obtained an R-square (R2) with 0.541. The model considers various factors, including the prompts used for generation; coherence, sentiment, and similarity in the conversation; and users' perceived dialog agents' favorability. We then further explore the effectiveness of the subset of our proposed model. The results showed that users' favorability and coherence, sentiment, and similarity in the dialogue are positive predictors of users' chat experience. Moreover, we found users may prefer dialog agents with characteristics of Extroversion, Openness, Conscientiousness, Agreeableness, and Non-Neuroticism. Through our research, an adaptive dialog system might use collected data to infer factors in our model, predict the chat experience for users through these factors, and optimize it by adjusting prompts.