When Should a Decision Maker Ignore the Advice of a Decision Aid?
This addresses the risk of over-reliance on imperfect AI tools for decision-makers, though it is incremental as it builds on existing human-computer interaction concepts.
The paper tackles the problem of decision aids based on fallible algorithms leading to worse performance than unaided decision-making, showing that interactive use can be counterproductive unless specific conditions are met.
This paper argues that the principal difference between decision aids and most other types of information systems is the greater reliance of decision aids on fallible algorithms--algorithms that sometimes generate incorrect advice. It is shown that interactive problem solving with a decision aid that is based on a fallible algorithm can easily result in aided performance which is poorer than unaided performance, even if the algorithm, by itself, performs significantly better than the unaided decision maker. This suggests that unless certain conditions are satisfied, using a decision aid as an aid is counterproductive. Some conditions under which a decision aid is best used as an aid are derived.