Connecting Humanities and Social Sciences: Applying Language and Speech Technology to Online Panel Surveys
This work addresses the challenge of analyzing open-ended survey responses for researchers in humanities and social sciences, offering a method to reduce manual effort, though it is incremental in applying existing technologies to a new domain.
The paper applied automatic speech recognition and transformer-based models to analyze spoken and typed open-ended responses in a Dutch panel survey, finding that ASR errors impact downstream analyses and that pre-trained models like BERT can perform sentiment analysis and topic modeling without extensive training on survey data.
In this paper, we explore the application of language and speech technology to open-ended questions in a Dutch panel survey. In an experimental wave respondents could choose to answer open questions via speech or keyboard. Automatic speech recognition (ASR) was used to process spoken responses. We evaluated answers from these input modalities to investigate differences between spoken and typed answers.We report the errors the ASR system produces and investigate the impact of these errors on downstream analyses. Open-ended questions give more freedom to answer for respondents, but entail a non-trivial amount of work to analyse. We evaluated the feasibility of using transformer-based models (e.g. BERT) to apply sentiment analysis and topic modelling on the answers of open questions. A big advantage of transformer-based models is that they are trained on a large amount of language materials and do not necessarily need training on the target materials. This is especially advantageous for survey data, which does not contain a lot of text materials. We tested the quality of automatic sentiment analysis by comparing automatic labeling with three human raters and tested the robustness of topic modelling by comparing the generated models based on automatic and manually transcribed spoken answers.