PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teams
For practitioners deploying human-AI teams in classification tasks, PLACO offers a cost-effective method to improve performance beyond either alone.
PLACO introduces a multi-stage framework for combining human and AI predictions in classification tasks, achieving up to 15% improvement in accuracy over prior methods while reducing human effort by 30%.
Human-AI teams play a pivotal role in improving overall system performance when neither the human nor the model can achieve such performance on their own. With the advent of powerful and accessible Generative AI models, several mundane tasks have morphed into Human-AI team tasks. From writing essays to developing advanced algorithms, humans have found that using AI assistance has led to an accelerated work pace like never before. In classification tasks, where the final output is a single hard label, it is crucial to address the combination of human and model output. Prior work elegantly solves this problem using Bayes rule, using the assumption that human and model output are conditionally independent given the ground truth. Specifically, it discusses a combination method to combine a single deterministic labeler (the human) and a probabilistic labeler (the classifier model) using the model's instance-level and the human's class-level calibrated probabilities.