Evaluating how interactive visualizations can assist in finding samples where and how computer vision models make mistakes
This addresses the issue of model opacity and complexity for end-users building and improving computer vision models, but it is incremental as it builds on existing interactive ML perspectives.
The paper tackled the problem of helping users identify and select images where computer vision models make mistakes, by evaluating two interactive visualizations in the Sprite system. The result showed that users with visualizations found more images across a wider set of potential error types compared to a baseline using a query language.
Creating Computer Vision (CV) models remains a complex practice, despite their ubiquity. Access to data, the requirement for ML expertise, and model opacity are just a few points of complexity that limit the ability of end-users to build, inspect, and improve these models. Interactive ML perspectives have helped address some of these issues by considering a teacher in the loop where planning, teaching, and evaluating tasks take place. We present and evaluate two interactive visualizations in the context of Sprite, a system for creating CV classification and detection models for images originating from videos. We study how these visualizations help Sprite's users identify (evaluate) and select (plan) images where a model is struggling and can lead to improved performance, compared to a baseline condition where users used a query language. We found that users who had used the visualizations found more images across a wider set of potential types of model errors.