Written Justifications are Key to Aggregate Crowdsourced Forecasts
This work addresses forecasting accuracy for crowdsourcing platforms, but it is incremental as it builds on existing aggregation methods with a focus on justification analysis.
The paper tackled the problem of improving crowdsourced forecast aggregation by modeling written justifications, finding that justifications are beneficial except in the final quarter of a question's lifecycle, with baselines like majority and weighted votes remaining competitive.
This paper demonstrates that aggregating crowdsourced forecasts benefits from modeling the written justifications provided by forecasters. Our experiments show that the majority and weighted vote baselines are competitive, and that the written justifications are beneficial to call a question throughout its life except in the last quarter. We also conduct an error analysis shedding light into the characteristics that make a justification unreliable.