RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Preview
This work addresses the need for failure-aware and controllable human-AI model editing in high-stakes domains like healthcare, though it is an incremental step combining existing concepts.
RuleEdit is an interactive system for human-AI model editing that surfaces likely failures via rule tables and provides prospective previews of edits. In a stroke rehabilitation assessment study, it improved Human+AI performance by 14.16% and increased post-update local performance gains from 11.50% to 36.38%.
Despite the promise of AI to assist complex decisions, practitioners still lack ways to detect likely failures and inspect the consequences of model edits before committing them. We present RuleEdit, an interactive, rule-guided human-AI model editing system that (i) surfaces likely failures through interpretable mismatch signals from rule tables and (ii) supports user-authored rule feedback with prospective previews of projected performance changes and embedding shifts. We instantiate RuleEdit in stroke rehabilitation assessment and evaluate it with health professionals and students. Rule-guided failure detection significantly increased Human + AI performance by 14.16\% ($p<0.001$) while improving rejection of incorrect AI and reducing both over- and under- reliance as well as ChangedToWrong decisions. In addition, presenting prospective embedding previews improved participants' feedback for model adaptation, increasing post-update local performance gains from 11.50\% to 36.38\% after incorporating users' rule-based feedback ($p<0.001$). Our findings show that mismatch-based failure cues and prospective impact previews can support failure-aware human-AI model editing, while also revealing a local-global tradeoff: edits that help a specific case can degrade performance when transferred globally. We discuss implications of designing failure-aware and controllable human-AI systems.