ECHO: An Open Research Platform for Evaluation of Chat, Human Behavior, and Outcomes
This platform addresses the need for scalable and reproducible human-centered AI evaluations, benefiting researchers in information retrieval, HCI, and social sciences, though it is incremental as it builds on existing evaluation methodologies.
The researchers tackled the challenge of conducting reproducible, mixed-method studies of human interaction with conversational AI and search engines by developing ECHO, an open platform that integrates surveys, tasks, and evaluations into a unified framework, resulting in a tool that logs interaction traces and exports structured datasets for analysis.
ECHO (Evaluation of Chat, Human behavior, and Outcomes) is an open research platform designed to support reproducible, mixed-method studies of human interaction with both conversational AI systems and Web search engines. It enables researchers from varying disciplines to orchestrate end-to-end experimental workflows that integrate consent and background surveys, chat-based and search-based information-seeking sessions, writing or judgment tasks, and pre- and post-task evaluations within a unified, low-coding-load framework. ECHO logs fine-grained interaction traces and participant responses, and exports structured datasets for downstream analysis. By supporting both chat and search alongside flexible evaluation instruments, ECHO lowers technical barriers for studying learning, decision making, and user experience across different information access paradigms, empowering researchers from information retrieval, HCI, and the social sciences to conduct scalable and reproducible human-centered AI evaluations.