Some people aren't worth listening to: periodically retraining classifiers with feedback from a team of end users
This addresses the challenge of maintaining classifier accuracy in business settings with ongoing user feedback loops, though it is incremental as it builds on existing multi-agent feedback models.
The paper tackles the problem of unreliable user feedback in document classification systems by proposing a classifier that identifies and filters out unreliable users, improving performance in subsequent retraining iterations.
Document classification is ubiquitous in a business setting, but often the end users of a classifier are engaged in an ongoing feedback-retrain loop with the team that maintain it. We consider this feedback-retrain loop from a multi-agent point of view, considering the end users as autonomous agents that provide feedback on the labelled data provided by the classifier. This allows us to examine the effect on the classifier's performance of unreliable end users who provide incorrect feedback. We demonstrate a classifier that can learn which users tend to be unreliable, filtering their feedback out of the loop, thus improving performance in subsequent iterations.