A Framework for Evaluating the Retrieval Effectiveness of Search Engines
This work addresses the need for reliable evaluation methods for search engines with enhanced result presentations, applicable across domains like Web search, but it is incremental as it builds on existing retrieval effectiveness concepts.
The paper tackles the problem of evaluating next-generation search engines that present enriched results beyond traditional lists, proposing a theoretical framework that extends retrieval effectiveness tests and examines how test design influences study outcomes.
This chapter presents a theoretical framework for evaluating next generation search engines. We focus on search engines whose results presentation is enriched with additional information and does not merely present the usual list of 10 blue links, that is, of ten links to results, accompanied by a short description. While Web search is used as an example here, the framework can easily be applied to search engines in any other area. The framework not only addresses the results presentation, but also takes into account an extension of the general design of retrieval effectiveness tests. The chapter examines the ways in which this design might influence the results of such studies and how a reliable test is best designed.