Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications
This addresses the challenge of automating variable identification in scholarly texts for social science researchers, but it is incremental as no new methods outperformed existing baselines.
The paper presents the SV-Ident 2022 shared task, which tackled the problem of identifying survey variables in sentences from social science publications, with participants making 9 submissions but none improving on baseline systems.
In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident