A Distributed Automatic Domain-Specific Multi-Word Term Recognition Architecture using Spark Ecosystem
This work addresses the need for efficient term recognition in computational linguistics and NLP by offering a scalable solution, though it is incremental as it builds on existing distributed methods.
The authors tackled the problem of extracting domain-specific multi-word terms from large text corpora by proposing a distributed Spark-based architecture, achieving improved accuracy and scalability as demonstrated on two real-world datasets.
Automatic Term Recognition is used to extract domain-specific terms that belong to a given domain. In order to be accurate, these corpus and language-dependent methods require large volumes of textual data that need to be processed to extract candidate terms that are afterward scored according to a given metric. To improve text preprocessing and candidate terms extraction and scoring, we propose a distributed Spark-based architecture to automatically extract domain-specific terms. The main contributions are as follows: (1) propose a novel distributed automatic domain-specific multi-word term recognition architecture built on top of the Spark ecosystem; (2) perform an in-depth analysis of our architecture in terms of accuracy and scalability; (3) design an easy-to-integrate Python implementation that enables the use of Big Data processing in fields such as Computational Linguistics and Natural Language Processing. We prove empirically the feasibility of our architecture by performing experiments on two real-world datasets.