More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice
This addresses the problem of designing effective multi-AI advice systems to balance accuracy and conformity for users, with incremental insights on panel dynamics.
The study investigated how consulting multiple AI advisors affects human decision-making, finding that small panels improved accuracy over a single AI, but larger panels did not, and that high consensus led to overreliance while dissent reduced conformity pressure.
Just as people improve decision-making by consulting diverse human advisors, they can now also consult with multiple AI systems. Prior work on group decision-making shows that advice aggregation creates pressure to conform, leading to overreliance. However, the conditions under which multi-AI consultation improves or undermines human decision-making remain unclear. We conducted experiments with three tasks in which participants received advice from panels of AIs. We varied panel size, within-panel consensus, and the human-likeness of presentation. Accuracy improved for small panels relative to a single AI; larger panels yielded no gains. The level of within-panel consensus affected participants' reliance on AI advice: High consensus fostered overreliance; a single dissent reduced pressure to conform; wide disagreement created confusion and undermined appropriate reliance. Human-like presentations increased perceived usefulness and agency in certain tasks, without raising conformity pressure. These findings yield design implications for presenting multi-AI advice that preserve accuracy while mitigating conformity.