FlexiTerm: A more efficient implementation of flexible multi-word term recognition
This work addresses the scalability issue for researchers and practitioners using unsupervised term recognition in big data contexts, but it is incremental as it focuses on re-implementation rather than new methodology.
The paper tackled the inefficiency of the original Java implementation of FlexiTerm, an unsupervised method for multi-word term recognition, by re-implementing it in Python, resulting in major efficiency improvements that enable its use in production-grade applications.
Terms are linguistic signifiers of domain-specific concepts. Automated recognition of multi-word terms (MWT) in free text is a sequence labelling problem, which is commonly addressed using supervised machine learning methods. Their need for manual annotation of training data makes it difficult to port such methods across domains. FlexiTerm, on the other hand, is a fully unsupervised method for MWT recognition from domain-specific corpora. Originally implemented in Java as a proof of concept, it did not scale well, thus offering little practical value in the context of big data. In this paper, we describe its re-implementation in Python and compare the performance of these two implementations. The results demonstrated major improvements in terms of efficiency, which allow FlexiTerm to transition from the proof of concept to the production-grade application.