AI-Assisted Conversational Interviewing: Effects on Data Quality and User Experience
This work addresses the problem of enhancing data quality in web surveys for researchers, though it is incremental as it builds on existing AI methods without major breakthroughs.
The study tackled the trade-off between scalable surveys and in-depth conversational interviews by introducing an AI-assisted conversational interviewing framework using LLMs, finding that AI chatbots moderately improved response detail and coding accuracy but slightly reduced respondent experience.
Standardized surveys scale efficiently but sacrifice depth, while conversational interviews improve response quality at the cost of scalability and consistency. This study bridges the gap between these methods by introducing a framework for AI-assisted conversational interviewing. To evaluate this framework, we conducted a web survey experiment where 1,800 participants were randomly assigned to AI 'chatbots' which use large language models (LLMs) to dynamically probe respondents for elaboration and interactively code open-ended responses to fixed questions developed by human researchers. We assessed the AI chatbot's performance in terms of coding accuracy, response quality, and respondent experience. Our findings reveal that AI chatbots perform moderately well in live coding even without survey-specific fine-tuning, despite slightly inflated false positive errors due to respondent acquiescence bias. Open-ended responses were more detailed and informative, but this came at a slight cost to respondent experience. Our findings highlight the feasibility of using AI methods such as chatbots enhanced by LLMs to enhance open-ended data collection in web surveys.