From Task Executors to Research Partners: Evaluating AI Co-Pilots Through Workflow Integration in Biomedical Research
This work addresses the problem of inadequate evaluation frameworks for AI co-pilots in biomedical research, which is incremental as it builds on existing benchmarking by identifying gaps and proposing improvements.
This rapid review analyzed benchmarking practices for AI systems in preclinical biomedical research, finding that all 14 identified benchmarks assess isolated component capabilities like data analysis and hypothesis generation, but fail to evaluate integrated workflows needed for authentic research collaboration. The authors propose a process-oriented evaluation framework addressing four dimensions—dialogue quality, workflow orchestration, session continuity, and researcher experience—to better assess AI as research co-pilots.
Artificial intelligence systems are increasingly deployed in biomedical research. However, current evaluation frameworks may inadequately assess their effectiveness as research collaborators. This rapid review examines benchmarking practices for AI systems in preclinical biomedical research. Three major databases and two preprint servers were searched from January 1, 2018 to October 31, 2025, identifying 14 benchmarks that assess AI capabilities in literature understanding, experimental design, and hypothesis generation. The results revealed that all current benchmarks assess isolated component capabilities, including data analysis quality, hypothesis validity, and experimental protocol design. However, authentic research collaboration requires integrated workflows spanning multiple sessions, with contextual memory, adaptive dialogue, and constraint propagation. This gap implies that systems excelling on component benchmarks may fail as practical research co-pilots. A process-oriented evaluation framework is proposed that addresses four critical dimensions absent from current benchmarks: dialogue quality, workflow orchestration, session continuity, and researcher experience. These dimensions are essential for evaluating AI systems as research co-pilots rather than as isolated task executors.