Joint Multi-Dimensional Model for Global and Time-Series Annotations
This work addresses the challenge of improving annotation fusion for multi-dimensional constructs like affect in crowdsourcing, which is incremental as it builds on existing methods by incorporating joint modeling.
The authors tackled the problem of fusing multi-dimensional crowdsourced annotations by proposing a generative model that jointly models dimensions, leading to more accurate ground truth estimates, with evaluations on synthetic data, real emotion corpora, and human annotations.
Crowdsourcing is a popular approach to collect annotations for unlabeled data instances. It involves collecting a large number of annotations from several, often naive untrained annotators for each data instance which are then combined to estimate the ground truth. Further, annotations for constructs such as affect are often multi-dimensional with annotators rating multiple dimensions, such as valence and arousal, for each instance. Most annotation fusion schemes however ignore this aspect and model each dimension separately. In this work we address this by proposing a generative model for multi-dimensional annotation fusion, which models the dimensions jointly leading to more accurate ground truth estimates. The model we propose is applicable to both global and time series annotation fusion problems and treats the ground truth as a latent variable distorted by the annotators. The model parameters are estimated using the Expectation-Maximization algorithm and we evaluate its performance using synthetic data and real emotion corpora as well as on an artificial task with human annotations