Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus
This work addresses terminology extraction and definition quality for fashion domain researchers and practitioners, but it is incremental as it benchmarks existing tools on a new dataset.
The paper compared term extraction accuracy of SketchEngine, TBXTools, and ChatGPT on an English-Russian fashion corpus, finding that ChatGPT maintained or improved precision as term count increased while others suffered reduced precision, and it produced definitions with reasonable accuracy but occasional missing specifics or deviations.
This paper compares the accuracy of the terms extracted using SketchEngine, TBXTools and ChatGPT. In addition, it evaluates the quality of the definitions produced by ChatGPT for these terms. The research is carried out on a comparable corpus of fashion magazines written in English and Russian collected from the web. A gold standard for the fashion terminology was also developed by identifying web pages that can be harvested automatically and contain definitions of terms from the fashion domain in English and Russian. This gold standard was used to evaluate the quality of the extracted terms and of the definitions produced. Our evaluation shows that TBXTools and SketchEngine, while capable of high recall, suffer from reduced precision as the number of terms increases, which affects their overall performance. Conversely, ChatGPT demonstrates superior performance, maintaining or improving precision as more terms are considered. Analysis of the definitions produced by ChatGPT for 60 commonly used terms in English and Russian shows that ChatGPT maintains a reasonable level of accuracy and fidelity across languages, but sometimes the definitions in both languages miss crucial specifics and include unnecessary deviations. Our research reveals that no single tool excels universally; each has strengths suited to particular aspects of terminology extraction and application.