Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research
This highlights transparency issues in NLP research, which could hinder reproducibility and accessibility for the community.
The study analyzed NLP research papers to quantify openness and its benefits, finding that over 30% of papers do not release promised artefacts and there is significant language-wise disparity in available artefacts.
We analysed a sample of NLP research papers archived in ACL Anthology as an attempt to quantify the degree of openness and the benefit of such an open culture in the NLP community. We observe that papers published in different NLP venues show different patterns related to artefact reuse. We also note that more than 30% of the papers we analysed do not release their artefacts publicly, despite promising to do so. Further, we observe a wide language-wise disparity in publicly available NLP-related artefacts.