Crowdsourcing the Perception of Machine Teaching
This work addresses the challenge of making machine learning systems more accessible to end-users through teachable interfaces, though it is incremental in exploring user perceptions rather than advancing technical methods.
The study investigated how non-experts conceptualize and perform machine teaching using a mobile testbed on Amazon Mechanical Turk, finding that participants incorporated diversity in training examples similar to human object recognition but often held misconceptions about model consistency and reasoning, with most not changing strategies on a second attempt.
Teachable interfaces can empower end-users to attune machine learning systems to their idiosyncratic characteristics and environment by explicitly providing pertinent training examples. While facilitating control, their effectiveness can be hindered by the lack of expertise or misconceptions. We investigate how users may conceptualize, experience, and reflect on their engagement in machine teaching by deploying a mobile teachable testbed in Amazon Mechanical Turk. Using a performance-based payment scheme, Mechanical Turkers (N = 100) are called to train, test, and re-train a robust recognition model in real-time with a few snapshots taken in their environment. We find that participants incorporate diversity in their examples drawing from parallels to how humans recognize objects independent of size, viewpoint, location, and illumination. Many of their misconceptions relate to consistency and model capabilities for reasoning. With limited variation and edge cases in testing, the majority of them do not change strategies on a second training attempt.