Evaluating Information Retrieval Systems for Kids
This is an incremental position paper that identifies evaluation gaps for IRS targeting children, a largely ignored user group.
The paper addresses the lack of evaluation methods for information retrieval systems designed for children, highlighting challenges such as missing frameworks, data limitations, and poor understanding of user judgments.
Evaluation of information retrieval systems (IRS) is a prominent topic among information retrieval researchers--mainly directed at a general population. Children require unique IRS and by extension different ways to evaluate these systems, but as a large population that use IRS have largely been ignored on the evaluation front. In this position paper, we explore many perspectives that must be considered when evaluating IRS; we specially discuss problems faced by researchers who work with children IRS, including lack of evaluation frameworks, limitations of data, and lack of user judgment understanding.