A Pitfall of Learning from User-generated Data: In-depth Analysis of Subjective Class Problem
This addresses a critical issue for data mining practitioners working with user-generated data, offering a practical solution to improve label reliability, though it is incremental in building on existing supervised learning challenges.
The paper tackles the problem of unreliable user-generated labels in supervised learning by distinguishing between subjective and objective classes, showing that objective classes are as reliable as expert labels while subjective ones are prone to bias and manipulation. It provides a framework to detect subjective labels without needing an oracle, helping practitioners avoid wasted resources.
Research in the supervised learning algorithms field implicitly assumes that training data is labeled by domain experts or at least semi-professional labelers accessible through crowdsourcing services like Amazon Mechanical Turk. With the advent of the Internet, data has become abundant and a large number of machine learning based systems started being trained with user-generated data, using categorical data as true labels. However, little work has been done in the area of supervised learning with user-defined labels where users are not necessarily experts and might be motivated to provide incorrect labels in order to improve their own utility from the system. In this article, we propose two types of classes in user-defined labels: subjective class and objective class - showing that the objective classes are as reliable as if they were provided by domain experts, whereas the subjective classes are subject to bias and manipulation by the user. We define this as a subjective class issue and provide a framework for detecting subjective labels in a dataset without querying oracle. Using this framework, data mining practitioners can detect a subjective class at an early stage of their projects, and avoid wasting their precious time and resources by dealing with subjective class problem with traditional machine learning techniques.