SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations
This work addresses the need for reproducible NLP tasks in non-English social media to detect subtle critical interventions, though it is incremental as it builds on existing methods for a new dataset.
The authors tackled the problem of detecting critical interventions in online conversations by introducing SPOT, an annotated French corpus, and found that fine-tuned encoder models outperformed prompted LLMs by over 10 percentage points in F1 score, with contextual metadata improving encoder F1 from 0.75 to 0.78.
We introduce SPOT (Stopping Points in Online Threads), the first annotated corpus translating the sociological concept of stopping point into a reproducible NLP task. Stopping points are ordinary critical interventions that pause or redirect online discussions through a range of forms (irony, subtle doubt or fragmentary arguments) that frameworks like counterspeech or social correction often overlook. We operationalize this concept as a binary classification task and provide reliable annotation guidelines. The corpus contains 43,305 manually annotated French Facebook comments linked to URLs flagged as false information by social media users, enriched with contextual metadata (article, post, parent comment, page or group, and source). We benchmark fine-tuned encoder models (CamemBERT) and instruction-tuned LLMs under various prompting strategies. Results show that fine-tuned encoders outperform prompted LLMs in F1 score by more than 10 percentage points, confirming the importance of supervised learning for emerging non-English social media tasks. Incorporating contextual metadata further improves encoder models F1 scores from 0.75 to 0.78. We release the anonymized dataset, along with the annotation guidelines and code in our code repository, to foster transparency and reproducible research.