Best Preprocessing Techniques for Sentiment Analysis
For practitioners in sentiment analysis, this paper offers a systematic, evidence-based ordering of preprocessing steps to avoid costly trial-and-error, though the findings are incremental and domain-specific.
This paper systematically examines the order of preprocessing techniques for sentiment analysis on Twitter data, finding that tokenisation is the most impactful and spelling correction the least. The optimal order is tokenisation, text cleaning, stemming, then stopword removal, providing a systematic approach to improve model output.
Sentiment analysis in Twitter datasets is important because it enables monitoring public opinion on products and analysis of political and social movements. One critical step is preprocessing: the automated processing of text for machine learning algorithms. Preprocessing plays a critical role in reducing noise and improving efficiency. However, little research has systematically examined the order in which preprocessing techniques are implemented. We find that, when accounting for order, spelling correction is the least impactful preprocessing technique, whereas tokenisation is the most impactful. Stemming and stop-word removal are interchangeable, and it is better to remove stop words without removing negation. The best order for applying the preprocessing techniques was tokenisation, text cleaning, stemming, and then stopword removal. Our results provide a systematic approach for practitioners to deploy preprocessing to improve model output without the costly preprocessing exploratory phase.