Trucks Don't Mean Trump: Diagnosing Human Error in Image Analysis
This work addresses the problem of reducing human error in high-stakes image interpretation tasks, such as political analysis, with potential applications to algorithmic systems, though it is incremental in applying existing error decomposition methods to a new domain.
The researchers tackled the problem of diagnosing human error in image analysis by developing machine learning methods to decompose errors into bias, variance, and noise terms, using a dataset of over 16 million human predictions about neighborhood voting patterns from Google Street View images. They identified specific misleading features like pickup trucks and demonstrated that their approach can improve accuracy and fairness in human-in-the-loop decision-making.
Algorithms provide powerful tools for detecting and dissecting human bias and error. Here, we develop machine learning methods to to analyze how humans err in a particular high-stakes task: image interpretation. We leverage a unique dataset of 16,135,392 human predictions of whether a neighborhood voted for Donald Trump or Joe Biden in the 2020 US election, based on a Google Street View image. We show that by training a machine learning estimator of the Bayes optimal decision for each image, we can provide an actionable decomposition of human error into bias, variance, and noise terms, and further identify specific features (like pickup trucks) which lead humans astray. Our methods can be applied to ensure that human-in-the-loop decision-making is accurate and fair and are also applicable to black-box algorithmic systems.