Exploration and Evaluation of Bias in Cyberbullying Detection with Machine Learning
It addresses bias issues in cyberbullying detection for researchers and practitioners, but is incremental as it focuses on evaluating existing methods rather than introducing new ones.
This study investigated how data collection and labeling biases affect machine learning models for cyberbullying detection, finding that models experience an average drop of 0.222 in Macro F1 Score when tested on unseen datasets.
It is well known that the usefulness of a machine learning model is due to its ability to generalize to unseen data. This study uses three popular cyberbullying datasets to explore the effects of data, how it's collected, and how it's labeled, on the resulting machine learning models. The bias introduced from differing definitions of cyberbullying and from data collection is discussed in detail. An emphasis is made on the impact of dataset expansion methods, which utilize current data points to fetch and label new ones. Furthermore, explicit testing is performed to evaluate the ability of a model to generalize to unseen datasets through cross-dataset evaluation. As hypothesized, the models have a significant drop in the Macro F1 Score, with an average drop of 0.222. As such, this study effectively highlights the importance of dataset curation and cross-dataset testing for creating models with real-world applicability. The experiments and other code can be found at https://github.com/rootdrew27/cyberbullying-ml.