Experiments on Crowdsourcing Policy Assessment
This work addresses the problem of resource-intensive policy evaluation for governments and organizations by exploring crowdsourcing as a cost-effective alternative, though it is incremental in testing existing methods on new data.
The study investigated whether non-expert crowds from virtual labor markets can replicate expert assessments of climate change adaptation policies, finding that crowds performed comparably to experts, with geographic relevance not significantly altering performance.
Can Crowds serve as useful allies in policy design? How do non-expert Crowds perform relative to experts in the assessment of policy measures? Does the geographic location of non-expert Crowds, with relevance to the policy context, alter the performance of non-experts Crowds in the assessment of policy measures? In this work, we investigate these questions by undertaking experiments designed to replicate expert policy assessments with non-expert Crowds recruited from Virtual Labor Markets. We use a set of ninety-six climate change adaptation policy measures previously evaluated by experts in the Netherlands as our control condition to conduct experiments using two discrete sets of non-expert Crowds recruited from Virtual Labor Markets. We vary the composition of our non-expert Crowds along two conditions: participants recruited from a geographical location directly relevant to the policy context and participants recruited at-large. We discuss our research methods in detail and provide the findings of our experiments.