Evaluating Human-AI Collaboration: A Review and Methodological Framework
This provides a methodological framework for researchers and practitioners to systematically evaluate HAIC, addressing a critical gap in real-world applications.
The paper tackles the challenge of evaluating Human-AI Collaboration (HAIC) systems by analyzing existing approaches and developing a new framework with a structured decision tree to select metrics based on HAIC modes, aiming to improve assessment of impact and success across domains like manufacturing and healthcare.
The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.