Personalization and Evaluation of Conversational Information Access
For researchers building conversational AI systems, this work provides novel methods for personal context extraction, personalized response generation, and automatic evaluation, though the contributions are incremental in nature.
This thesis addresses personalization and evaluation in Conversational Information Access (CIA) by introducing methods for entity linking in conversations (CREL), personalized response generation (LAPS), and automatic evaluation (FACE). CREL achieves state-of-the-art performance on the ConEL dataset, LAPS enables efficient construction of personalized datasets, and FACE aligns closely with human judgments for conversation-level evaluation.
Conversational interactions have reshaped information retrieval systems, as users increasingly favour direct answers over traditional hyperlinks. To build reliable Conversational Information Access (CIA) systems that account for personal context, this thesis addresses challenges: (1) personal context extraction, (2) personalized response generation, and (3) effective and interpretable system evaluation. First, we tackle personal context extraction by studying what Entity Linking (EL) in conversations entails, introducing a dataset for conversational entity linking (ConEL), and proposing CREL, a novel EL method tailored for conversational settings. Second, we focus on personalized response generation by proposing LAPS, a method for efficiently constructing large-scale, human-written, personalized conversational datasets, and using them to study how users' preferences can be utilized to generate personalized responses. Finally, we address the need for effective and interpretable system evaluation by introducing FACE, an automatic, reference-free method that assesses entire conversations and aligns closely with human judgments.