The risk of sub-optimal use of Open Source NLP Software: UKB is inadvertently state-of-the-art in knowledge-based WSD
This addresses reproducibility and optimal use problems for NLP researchers and practitioners, but is incremental as it focuses on re-evaluating an existing tool.
The authors found that UKB, an open-source tool for knowledge-based Word Sense Disambiguation (WSD) released in 2009, has been used sub-optimally but is actually state-of-the-art in this task nine years later, highlighting issues with default settings and reproducibility instructions.
UKB is an open source collection of programs for performing, among other tasks, knowledge-based Word Sense Disambiguation (WSD). Since it was released in 2009 it has been often used out-of-the-box in sub-optimal settings. We show that nine years later it is the state-of-the-art on knowledge-based WSD. This case shows the pitfalls of releasing open source NLP software without optimal default settings and precise instructions for reproducibility.