Vietnamese Word Segmentation with SVM: Ambiguity Reduction and Suffix Capture
This work addresses word segmentation for Vietnamese language processing, offering incremental improvements over existing methods.
The paper tackles Vietnamese word segmentation by using an SVM classifier with novel features to reduce ambiguity and capture suffixes, achieving a higher F1-score than state-of-the-art methods like UETsegmenter and RDRsegmenter on benchmark datasets.
In this paper, we approach Vietnamese word segmentation as a binary classification by using the Support Vector Machine classifier. We inherit features from prior works such as n-gram of syllables, n-gram of syllable types, and checking conjunction of adjacent syllables in the dictionary. We propose two novel ways to feature extraction, one to reduce the overlap ambiguity and the other to increase the ability to predict unknown words containing suffixes. Different from UETsegmenter and RDRsegmenter, two state-of-the-art Vietnamese word segmentation methods, we do not employ the longest matching algorithm as an initial processing step or any post-processing technique. According to experimental results on benchmark Vietnamese datasets, our proposed method obtained a better F1-score than the prior state-of-the-art methods UETsegmenter, and RDRsegmenter.