Segmenting Natural Language Sentences via Lexical Unit Analysis
This work provides a general and efficient framework for sequence segmentation, offering improved performance for various natural language processing tasks, particularly for identifying longer segments.
This paper introduces Lexical Unit Analysis (LUA), a framework for sequence segmentation that uses dynamic programming to find the highest-scoring valid segmentation of a natural language sentence. LUA achieved state-of-the-art performance on 13 out of 15 datasets across five tasks, with notable improvements in F1 scores for long-length segments.
In this work, we present Lexical Unit Analysis (LUA), a framework for general sequence segmentation tasks. Given a natural language sentence, LUA scores all the valid segmentation candidates and utilizes dynamic programming (DP) to extract the maximum scoring one. LUA enjoys a number of appealing properties such as inherently guaranteeing the predicted segmentation to be valid and facilitating globally optimal training and inference. Besides, the practical time complexity of LUA can be reduced to linear time, which is very efficient. We have conducted extensive experiments on 5 tasks, including syntactic chunking, named entity recognition (NER), slot filling, Chinese word segmentation, and Chinese part-of-speech (POS) tagging, across 15 datasets. Our models have achieved the state-of-the-art performances on 13 of them. The results also show that the F1 score of identifying long-length segments is notably improved.