Gonzalo Ramos

Priyan Vaithilingam, Elena L. Glassman, Nathalie Henry Riche et al.

People working with data often move their data across multiple applications, because they rely on these apps' complementing user experiences to best complete their tasks. Since traditional copy-and-paste approaches do not accommodate diverse table representations adopted by different apps, users spend considerable effort to reconstruct data formats and visual representations, making cross-app workflows costly. For example, when transferring a spreadsheet table with conditional formatting to a markup document, users spend substantial time translating its structure into appropriate tags and manually reformat color. This paper introduces MagicCopy, an AI-powered cross-app copy-and-paste, leveraging source and target contexts and user-specified instructions in natural language to automatically extract, parse, transform, and (re)format data from one app to another. In a study with sixteen participants, users quickly learned and applied MagicCopy to move data across three pairs of tools. Participants further explored diverse applications of MagicCopy to support more streamlined crossed-application interaction in their workflows.

Hayeong Song, Gonzalo Ramos, Peter Bodik

Creating Computer Vision (CV) models remains a complex practice, despite their ubiquity. Access to data, the requirement for ML expertise, and model opacity are just a few points of complexity that limit the ability of end-users to build, inspect, and improve these models. Interactive ML perspectives have helped address some of these issues by considering a teacher in the loop where planning, teaching, and evaluating tasks take place. We present and evaluate two interactive visualizations in the context of Sprite, a system for creating CV classification and detection models for images originating from videos. We study how these visualizations help Sprite's users identify (evaluate) and select (plan) images where a model is struggling and can lead to improved performance, compared to a baseline condition where users used a query language. We found that users who had used the visualizations found more images across a wider set of potential types of model errors.

Ahmed Elgohary, Christopher Meek, Matthew Richardson et al.

We study semantic parsing in an interactive setting in which users correct errors with natural language feedback. We present NL-EDIT, a model for interpreting natural language feedback in the interaction context to generate a sequence of edits that can be applied to the initial parse to correct its errors. We show that NL-EDIT can boost the accuracy of existing text-to-SQL parsers by up to 20% with only one turn of correction. We analyze the limitations of the model and discuss directions for improvement and evaluation. The code and datasets used in this paper are publicly available at http://aka.ms/NLEdit.