Validating Political Position Predictions of Arguments
This work addresses the problem of reliable validation for subjective continuous knowledge in domains like political discourse, enabling improved graph-based reasoning and retrieval-augmented generation, though it is incremental in refining evaluation methods.
The paper tackled the challenge of validating subjective, continuous attributes like political positions in knowledge representation by proposing a dual-scale validation framework combining pointwise and pairwise human annotation for political stance prediction in arguments. The result showed moderate pointwise human-model agreement (Krippendorff's α=0.578) and substantially stronger pairwise alignment (α=0.86 for the best model) using 22 language models on 23,228 arguments from 30 debates.
Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation. We address this challenge through a dual-scale validation framework applied to political stance prediction in argumentative discourse, combining pointwise and pairwise human annotation. Using 22 language models, we construct a large-scale knowledge base of political position predictions for 23,228 arguments drawn from 30 debates that appeared on the UK politicial television programme \textit{Question Time}. Pointwise evaluation shows moderate human-model agreement (Krippendorff's $α=0.578$), reflecting intrinsic subjectivity, while pairwise validation reveals substantially stronger alignment between human- and model-derived rankings ($α=0.86$ for the best model). This work contributes: (i) a practical validation methodology for subjective continuous knowledge that balances scalability with reliability; (ii) a validated structured argumentation knowledge base enabling graph-based reasoning and retrieval-augmented generation in political domains; and (iii) evidence that ordinal structure can be extracted from pointwise language models predictions from inherently subjective real-world discourse, advancing knowledge representation capabilities for domains where traditional symbolic or categorical approaches are insufficient.