A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement
This addresses the challenge of understanding annotation inconsistencies in data labeling for researchers, though it appears incremental as it builds on existing decomposition concepts.
The paper tackles the problem of analyzing inter-annotator disagreement by proposing a method to decompose annotations into separate levels, applied to a narrative analysis dataset to test hypotheses about disagreement sources and reveal latent structures.
We propose a novel method to conceptually decompose an existing annotation into separate levels, allowing the analysis of inter-annotators disagreement in each level separately. We suggest two distinct strategies in order to actualize this approach: a theoretically-driven one, in which the researcher defines a decomposition based on prior knowledge of the annotation task, and an exploration-based one, in which many possible decompositions are inductively computed and presented to the researcher for interpretation and evaluation. Utilizing a recently constructed dataset for narrative analysis as our use-case, we apply each of the two strategies to demonstrate the potential of our approach in testing hypotheses regarding the sources of annotation disagreements, as well as revealing latent structures and relations within the annotation task. We conclude by suggesting how to extend and generalize our approach, as well as use it for other purposes.