LSICC: A Large Scale Informal Chinese Corpus
This provides a domain-specific resource for training NLP models on informal Chinese text, but it is incremental as it focuses on data collection rather than methodological innovation.
The authors tackled the gap between real-world tasks and existing Chinese corpora by introducing a large-scale informal Chinese corpus containing 37 million book reviews and 50,000 netizen comments, and they explored informal word frequencies to show differences from existing datasets.
Deep learning based natural language processing model is proven powerful, but need large-scale dataset. Due to the significant gap between the real-world tasks and existing Chinese corpus, in this paper, we introduce a large-scale corpus of informal Chinese. This corpus contains around 37 million book reviews and 50 thousand netizen's comments to the news. We explore the informal words frequencies of the corpus and show the difference between our corpus and the existing ones. The corpus can be further used to train deep learning based natural language processing tasks such as Chinese word segmentation, sentiment analysis.