Role of Intrinsic Motivation in User Interface Design to Enhance Worker Performance in Amazon MTurk
This addresses the challenge of obtaining high-quality crowdsourced annotations for training machine learning models in marine biology, though it is incremental as it applies existing psychological theory to a specific domain.
The paper tackled the problem of poor quality annotations from Amazon MTurk workers for marine life image datasets by designing a new interface based on Self-Determination Theory, resulting in a significant improvement in worker accuracy.
Biologists and scientists have been tackling the problem of marine life monitoring and fish stock estimation for many years now. Efforts are now directed to move towards non-intrusive methods, by utilizing specially designed underwater robots to collect images of the marine population. Training machine learning algorithms on the images collected, we can now estimate the population. This in turn helps to impose regulations to control overfishing. To train these models, however, we need annotated images. Annotation of large sets of images collected over a decade is quite challenging. Hence, we resort to Amazon Mechanical Turk (MTurk), a crowdsourcing platform, for the image annotation task. Although it is fast to get work done in MTurk, the work obtained is often of poor quality. This work aims to understand the human factors in designing Human Intelligence Tasks (HITs), from the perspective of the Self-Determination Theory. Applying elements from the theory, we design an HIT to increase the competence and motivation of the workers. Within our experimental framework, we find that the new interface significantly improves the accuracy of worker performance.