Is Evaluating Visual Search Interfaces in Digital Libraries Still an Issue?
This addresses a methodological problem for researchers and developers in digital libraries, but it is incremental as it builds on existing evaluation challenges without introducing new paradigms.
The paper argues that visual interfaces for digital libraries are rarely integrated due to a lack of comparability in evaluation results, proposing a common evaluation setup with a reference system and standardized tasks to address this issue.
Although various visual interfaces for digital libraries have been developed in prototypical systems, very few of these visual approaches have been integrated into today's digital libraries. In this position paper we argue that this is most likely due to the fact that the evaluation results of most visual systems lack comparability. There is no fix standard on how to evaluate visual interactive user interfaces. Therefore it is not possible to identify which approach is more suitable for a certain context. We feel that the comparability of evaluation results could be improved by building a common evaluation setup consisting of a reference system, based on a standardized corpus with fixed tasks and a panel for possible participants.