Personalization in Human-AI Teams: Improving the Compatibility-Accuracy Tradeoff
This addresses the problem of maintaining user satisfaction in human-AI teams when AI models are updated, offering a personalized approach that is incremental but with notable benefits for some users.
The paper tackles the trade-off between AI model accuracy and compatibility with user experience after updates, showing that personalizing the loss function can improve this trade-off for specific users. Results include moderate average improvements of around 20% and large gains up to 300% for certain users.
AI systems that model and interact with users can update their models over time to reflect new information and changes in the environment. Although these updates may improve the overall performance of the AI system, they may actually hurt the performance with respect to individual users. Prior work has studied the trade-off between improving the system's accuracy following an update and the compatibility of the updated system with prior user experience. The more the model is forced to be compatible with a prior version, the higher loss in accuracy it will incur. In this paper, we show that by personalizing the loss function to specific users, in some cases it is possible to improve the compatibility-accuracy trade-off with respect to these users (increase the compatibility of the model while sacrificing less accuracy). We present experimental results indicating that this approach provides moderate improvements on average (around 20%) but large improvements for certain users (up to 300%).