Users Mispredict Their Own Preferences for AI Writing Assistance
This research addresses a critical misalignment in user preferences for proactive AI writing systems, with direct implications for natural language generation, though it is incremental in nature.
The study tackled the problem of predicting when users want AI writing assistance by analyzing their preferences, finding that users mispredict their own preferences, with systems based on stated preferences achieving only 57.7% accuracy compared to 61.3% for those using behavioral patterns.
Proactive AI writing assistants need to predict when users want drafting help, yet we lack empirical understanding of what drives preferences. Through a factorial vignette study with 50 participants making 750 pairwise comparisons, we find compositional effort dominates decisions ($ρ= 0.597$) while urgency shows no predictive power ($ρ\approx 0$). More critically, users exhibit a striking perception-behavior gap: they rank urgency first in self-reports despite it being the weakest behavioral driver, representing a complete preference inversion. This misalignment has measurable consequences. Systems designed from users' stated preferences achieve only 57.7\% accuracy, underperforming even naive baselines, while systems using behavioral patterns reach significantly higher 61.3\% ($p < 0.05$). These findings demonstrate that relying on user introspection for system design actively misleads optimization, with direct implications for proactive natural language generation (NLG) systems.