Design and Evaluation of Camera-Centric Mobile Crowdsourcing Applications
This addresses the problem of optimizing data collection for computer vision tasks by balancing user effort and data utility, though it is incremental in design evaluation.
The study investigated how varying labeling effort in camera-based mobile crowdsourcing apps affects user contributions and data quality, finding that higher labeling requests did not reduce image collection or satisfaction and improved image retrieval performance.
The data that underlies automated methods in computer vision and machine learning, such as image retrieval and fine-grained recognition, often comes from crowdsourcing. In contexts that rely on the intrinsic motivation of users, we seek to understand how the application design affects a user's willingness to contribute and the quantity and quality of the data they capture. In this project, we designed three versions of a camera-based mobile crowdsourcing application, which varied in the amount of labeling effort requested of the user and conducted a user study to evaluate the trade-off between the level of user-contributed information requested and the quantity and quality of labeled images collected. The results suggest that higher levels of user labeling do not lead to reduced contribution. Users collected and annotated the most images using the application version with the highest requested level of labeling with no decrease in user satisfaction. In preliminary experiments, the additional labeled data supported increased performance on an image retrieval task.