New Perspectives in Sinographic Language Processing Through the Use of Character Structure
This work addresses text mining efficiency for sinographic languages like Chinese and Japanese, but it is incremental as it builds on existing methods with modest gains.
The paper tackled the problem of graphical variation in Chinese characters by defining allographic classes and using character structure to enhance text models, resulting in a 3% improvement in precision on a text classification task with an 18-million-character corpus.
Chinese characters have a complex and hierarchical graphical structure carrying both semantic and phonetic information. We use this structure to enhance the text model and obtain better results in standard NLP operations. First of all, to tackle the problem of graphical variation we define allographic classes of characters. Next, the relation of inclusion of a subcharacter in a characters, provides us with a directed graph of allographic classes. We provide this graph with two weights: semanticity (semantic relation between subcharacter and character) and phoneticity (phonetic relation) and calculate "most semantic subcharacter paths" for each character. Finally, adding the information contained in these paths to unigrams we claim to increase the efficiency of text mining methods. We evaluate our method on a text classification task on two corpora (Chinese and Japanese) of a total of 18 million characters and get an improvement of 3% on an already high baseline of 89.6% precision, obtained by a linear SVM classifier. Other possible applications and perspectives of the system are discussed.