Facilitating Longitudinal Interaction Studies of AI Systems
This work tackles challenges in conducting longer-term studies of AI systems for UIST researchers, but it is incremental as it focuses on community-building and workshop activities rather than new technical solutions.
The paper addresses the problem of insufficient one-time evaluations for AI systems by proposing a workshop to facilitate longitudinal interaction studies, aiming to prepare researchers with practical strategies for such research.
UIST researchers develop tools to address user challenges. However, user interactions with AI evolve over time through learning, adaptation, and repurposing, making one time evaluations insufficient. Capturing these dynamics requires longer-term studies, but challenges in deployment, evaluation design, and data collection have made such longitudinal research difficult to implement. Our workshop aims to tackle these challenges and prepare researchers with practical strategies for longitudinal studies. The workshop includes a keynote, panel discussions, and interactive breakout groups for discussion and hands-on protocol design and tool prototyping sessions. We seek to foster a community around longitudinal system research and promote it as a more embraced method for designing, building, and evaluating UIST tools.