Stance Labels Fail When They Matter Most: The Projection Problem in Stance Detection
This highlights a fundamental flaw in stance detection for social media analysis, where label reliability fails precisely in complex, high-stakes scenarios.
The paper identifies the projection problem in stance detection, where multi-dimensional attitudes are compressed into single labels, causing annotator disagreement when dimensions conflict; a pilot study on SemEval-2016 data shows label agreement drops to Krippendorff's α=0.085 while dimensional agreement rises to α=0.334 in such cases.
Stance detection is nearly always formulated as classifying text into Favor, Against, or Neutral -- a convention inherited from debate analysis and applied without modification to social media since SemEval-2016. But attitudes toward complex targets are not unitary: a person can accept climate science while opposing carbon taxes, expressing support on one dimension and opposition on another. When annotators must compress such multi-dimensional attitudes into a single label, different annotators weight different dimensions -- producing disagreement that reflects not confusion but different compression choices. We call this the \textbf{projection problem}, and show that its cost is conditional: when a text's dimensions align, any weighting yields the same label and three-way annotation works well; when dimensions conflict, label agreement collapses while agreement on individual dimensions remains intact. A pilot study on SemEval-2016 Task 6 confirms this crossover: on dimension-consistent texts, label agreement (Krippendorff's $α= 0.307$) exceeds dimensional agreement ($α= 0.082$); on dimension-conflicting texts, the pattern reverses -- label $α$ drops to $0.085$ while dimensional $α$ rises to $0.334$, with Policy reaching $0.572$. The projection problem is real -- but it activates precisely where it matters most.