ACM-CR: A Manually Annotated Test Collection for Citation Recommendation
This work addresses the need for reliable evaluation in citation recommendation for researchers, but it is incremental as it focuses on dataset creation and baseline testing.
The authors tackled the problem of noisy and unreliable test collections for citation recommendation by creating a manually annotated, publicly available test collection, and they evaluated content-based baseline models on it, providing initial results for future improvements.
Citation recommendation is intended to assist researchers in the process of searching for relevant papers to cite by recommending appropriate citations for a given input text. Existing test collections for this task are noisy and unreliable since they are built automatically from parsed PDF papers. In this paper, we present our ongoing effort at creating a publicly available, manually annotated test collection for citation recommendation. We also conduct a series of experiments to evaluate the effectiveness of content-based baseline models on the test collection, providing results for future work to improve upon. Our test collection and code to replicate experiments are available at https://github.com/boudinfl/acm-cr