Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses
This addresses the need for efficient quality control in marketing research by automating the evaluation of human-written survey responses, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the problem of automatically evaluating open-ended human survey responses to filter low-quality ones, proposing a two-stage framework that outperforms existing metrics and shows strong correlation with expert assessments.
Open-ended survey responses provide valuable insights in marketing research, but low-quality responses not only burden researchers with manual filtering but also risk leading to misleading conclusions, underscoring the need for effective evaluation. Existing automatic evaluation methods target LLM-generated text and inadequately assess human-written responses with their distinct characteristics. To address such characteristics, we propose a two-stage evaluation framework specifically designed for human survey responses. First, gibberish filtering removes nonsensical responses. Then, three dimensions-effort, relevance, and completeness-are evaluated using LLM capabilities, grounded in empirical analysis of real-world survey data. Validation on English and Korean datasets shows that our framework not only outperforms existing metrics but also demonstrates high practical applicability for real-world applications such as response quality prediction and response rejection, showing strong correlations with expert assessment.