SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research
This provides a standardized evaluation framework for researchers and practitioners in social media NLP, though it is incremental as it consolidates existing tasks.
The authors tackled the fragmented landscape of NLP for social media by introducing SuperTweetEval, a unified benchmark including heterogeneous tasks and datasets, and found that social media remains challenging for models despite recent advances.
Despite its relevance, the maturity of NLP for social media pales in comparison with general-purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.