Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts
This addresses the reliability of crowdsourced linguistic data for language learning applications, but it is incremental as it builds on prior work in crowdsourcing and expert-non-expert comparisons.
The study investigated whether experts and non-experts agree on ranking multi-word expressions by difficulty in a crowdsourcing experiment, finding that rankings from all three groups correlated highly, indicating professional insights do not influence judgments in comparative settings.
In this study we investigate to which degree experts and non-experts agree on questions of difficulty in a crowdsourcing experiment. We ask non-experts (second language learners of Swedish) and two groups of experts (teachers of Swedish as a second/foreign language and CEFR experts) to rank multi-word expressions in a crowdsourcing experiment. We find that the resulting rankings by all the three tested groups correlate to a very high degree, which suggests that judgments produced in a comparative setting are not influenced by professional insights into Swedish as a second language.