Dirk Lewandowski

Sebastian Schultheiß, Dirk Lewandowski

People have a high level of trust in search engines, especially Google, but only limited knowledge of them, as numerous studies have shown. This leads to the question: To what extent is this trust justified considering the lack of familiarity among users with how search engines work and the business models they are founded on? We assume that trust in Google, search engine preferences, and knowledge of result types are interrelated. To examine this assumption, we conducted a representative online survey with n = 2,012 German internet users. We show that users with little search engine knowledge are more likely to trust and use Google than users with more knowledge. A contradiction revealed itself - users strongly trust Google, yet they are unable to adequately evaluate search results. This may be problematic since it can potentially affect knowledge acquisition. Consequently, there is a need to promote user information literacy to create a more solid foundation for user trust in search engines.

Dirk Lewandowski

A proposal for building an index of the Web that separates the infrastructure part of the search engine - the index - from the services part that will form the basis for myriad search engines and other services utilizing Web data on top of a public infrastructure open to everyone.

Sebastian Suenkler, Dirk Lewandowski

The objective of this research was to find out how the two search engines Google and Bing perform when users work freely on pre-defined tasks, and judge the relevance of the results immediately after finishing their search session. In a user study, 64 participants conducted two search tasks each, and then judged the results on the following: (1) The quality of the results they selected in their search sessions, (2) The quality of the results they were presented with in their search sessions (but which they did not click on), (3) The quality of the results from the competing search engine for their queries (which they did not see in their search session). We found that users heavily relied on Google, that Google produced more relevant results than Bing, that users were well able to select relevant results from the results lists, and that users judged the relevance of results lower when they regarded a task as difficult and did not find the correct information.

Dirk Lewandowski, Friederike Kerkmann, Sandra Ruemmele et al.

This representative study of German search engine users (N=1,000) focuses on the ability of users to distinguish between organic results and advertisements on Google results pages. We combine questions about Google's business with task-based studies in which users were asked to distinguish between ads and organic results in screenshots of results pages. We find that only a small percentage of users is able to reliably distinguish between ads and organic results, and that user knowledge of Google's business model is very limited. We conclude that ads are insufficiently labelled as such, and that many users may click on ads assuming that they are selecting organic results.

Philipp Schaer, Philipp Mayr, Sebastian Sünkler et al.

Users of web search engines are known to mostly focus on the top ranked results of the search engine result page. While many studies support this well known information seeking pattern only few studies concentrate on the question what users are missing by neglecting lower ranked results. To learn more about the relevance distributions in the so-called long tail we conducted a relevance assessment study with the Million Short long-tail web search engine. While we see a clear difference in the content between the head and the tail of the search engine result list we see no statistical significant differences in the binary relevance judgments and weak significant differences when using graded relevance. The tail contains different but still valuable results. We argue that the long tail can be a rich source for the diversification of web search engine result lists but it needs more evaluation to clearly describe the differences.

Georg Singer, Pille Pruulmann-Vengerfeldt, Ulrich Norbisrath et al.

It is widely known that people become better at an activity if they perform this activity long and often. Yet, the question is whether being active in related areas like communicating online, writing blog articles or commenting on community forums have an impact on a persons ability to perform Web searches, is still unanswered. Web searching has become a key task conducted online; in this paper we present our findings on whether the user type, which categorises a persons online activities, has an impact on her or his search capabilities. We show (1) the characteristics of different user types when carrying out simple search tasks; (2) their characteristics when carrying out complex search tasks; and, (3) the significantly different user type characteristics between simple and complex search tasks. The results are based on an experiment with 56 ordinary Web users in a laboratory environment. The Search-Logger study framework was used to analyze and measure user behavior when carrying out a set of 12 predefined search tasks. Our findings include the fact that depending on task type (simple or complex) significant differences can be observed between users of different types.

Dirk Lewandowski

This chapter presents a theoretical framework for evaluating next generation search engines. We focus on search engines whose results presentation is enriched with additional information and does not merely present the usual list of 10 blue links, that is, of ten links to results, accompanied by a short description. While Web search is used as an example here, the framework can easily be applied to search engines in any other area. The framework not only addresses the results presentation, but also takes into account an extension of the general design of retrieval effectiveness tests. The chapter examines the ways in which this design might influence the results of such studies and how a reliable test is best designed.

Dirk Lewandowski

Purpose - To test major Web search engines on their performance on navigational queries, i.e. searches for homepages. Design/methodology/approach - 100 real user queries are posed to six search engines (Google, Yahoo, MSN, Ask, Seekport, and Exalead). Users described the desired pages, and the results position of these is recorded. Measured success N and mean reciprocal rank are calculated. Findings - Performance of the major search engines Google, Yahoo, and MSN is best, with around 90 percent of queries answered correctly. Ask and Exalead perform worse but receive good scores as well. Research limitations/implications - All queries were in German, and the German-language interfaces of the search engines were used. Therefore, the results are only valid for German queries. Practical implications - When designing a search engine to compete with the major search engines, care should be taken on the performance on navigational queries. Users can be influenced easily in their quality ratings of search engines based on this performance. Originality/value - This study systematically compares the major search engines on navigational queries and compares the findings with studies on the retrieval effectiveness of the engines on informational queries. Paper type - research paper

Dirk Lewandowski

We carried out a retrieval effectiveness test on the three major web search engines (i.e., Google, Microsoft and Yahoo). In addition to relevance judgments, we classified the results according to their commercial intent and whether or not they carried any advertising. We found that all search engines provide a large number of results with a commercial intent. Google provides significantly more commercial results than the other search engines do. However, the commercial intent of a result did not influence jurors in their relevance judgments.

Dirk Lewandowski

Purpose: This paper discusses ranking factors suitable for library materials and shows that ranking in general is a complex process and that ranking for library materials requires a variety of techniques. Design/methodology/approach: The relevant literature is reviewed to provide a systematic overview of suitable ranking factors. The discussion is based on an overview of ranking factors used in Web search engines. Findings: While there are a wide variety of ranking factors applicable to library materials, todays library systems use only some of them. When designing a ranking component for the library catalogue, an individual weighting of applicable factors is necessary. Research limitations/applications: While this article discusses different factors, no particular ranking formula is given. However, this article presents the argument that such a formula must always be individual to a certain use case. Practical implications: The factors presented can be considered when designing a ranking component for a librarys search system or when discussing such a project with an ILS vendor. Originality/value: This paper is original in that it is the first to systematically discuss ranking of library materials based on the main factors used by Web search engines.

Nadine Hoechstoetter, Dirk Lewandowski

This paper investigates the composition of search engine results pages. We define what elements the most popular web search engines use on their results pages (e.g., organic results, advertisements, shortcuts) and to which degree they are used for popular vs. rare queries. Therefore, we send 500 queries of both types to the major search engines Google, Yahoo, Live.com and Ask. We count how often the different elements are used by the individual engines. In total, our study is based on 42,758 elements. Findings include that search engines use quite different approaches to results pages composition and therefore, the user gets to see quite different results sets depending on the search engine and search query used. Organic results still play the major role in the results pages, but different shortcuts are of some importance, too. Regarding the frequency of certain host within the results sets, we find that all search engines show Wikipedia results quite often, while other hosts shown depend on the search engine used. Both Google and Yahoo prefer results from their own offerings (such as YouTube or Yahoo Answers). Since we used the .com interfaces of the search engines, results may not be valid for other country-specific interfaces.

Dirk Lewandowski

Purpose: To compare five major Web search engines (Google, Yahoo, MSN, Ask.com, and Seekport) for their retrieval effectiveness, taking into account not only the results but also the results descriptions. Design/Methodology/Approach: The study uses real-life queries. Results are made anonymous and are randomised. Results are judged by the persons posing the original queries. Findings: The two major search engines, Google and Yahoo, perform best, and there are no significant differences between them. Google delivers significantly more relevant result descriptions than any other search engine. This could be one reason for users perceiving this engine as superior. Research Limitations: The study is based on a user model where the user takes into account a certain amount of results rather systematically. This may not be the case in real life. Practical Implications: Implies that search engines should focus on relevant descriptions. Searchers are advised to use other search engines in addition to Google. Originality/Value: This is the first major study comparing results and descriptions systematically and proposes new retrieval measures to take into account results descriptions

Dirk Lewandowski

Purpose - To test the ability of major search engines, Google, Yahoo, MSN, and Ask, to distinguish between German and English-language documents Design/methodology/approach - 50 queries, using words common in German and in English, were posed to the engines. The advanced search option of language restriction was used, once in German and once in English. The first 20 results per engine in each language were investigated. Findings - While none of the search engines faces problems in providing results in the language of the interface that is used, both Google and MSN face problems when the results are restricted to a foreign language. Research limitations/implications - Search engines were only tested in German and in English. We have only anecdotal evidence that the problems are the same with other languages. Practical implications - Searchers should not use the language restriction in Google and MSN when searching for foreign-language documents. Instead, searchers should use Yahoo or Ask. If searching for foreign language documents in Google or MSN, the interface in the target language/country should be used. Value of paper - Demonstrates a problem with search engines that has not been previously investigated.

Dirk Lewandowski

The path to greater diversity, as we have seen, cannot be achieved by merely hoping for a new search engine nor will government support for a single alternative achieve this goal. What is instead required is to create the conditions that will make establishing such a search engine possible in the first place. I describe how building and maintaining a proprietary index is the greatest deterrent to such an undertaking. We must first overcome this obstacle. Doing so will still not solve the problem of the lack of diversity in the search engine marketplace. But it may establish the conditions necessary to achieve that desired end.

Dirk Lewandowski

Search engine retrieval effectiveness studies are usually small-scale, using only limited query samples. Furthermore, queries are selected by the researchers. We address these issues by taking a random representative sample of 1,000 informational and 1,000 navigational queries from a major German search engine and comparing Google's and Bing's results based on this sample. Jurors were found through crowdsourcing, data was collected using specialised software, the Relevance Assessment Tool (RAT). We found that while Google outperforms Bing in both query types, the difference in the performance for informational queries was rather low. However, for navigational queries, Google found the correct answer in 95.3 per cent of cases whereas Bing only found the correct answer 76.6 per cent of the time. We conclude that search engine performance on navigational queries is of great importance, as users in this case can clearly identify queries that have returned correct results. So, performance on this query type may contribute to explaining user satisfaction with search engines.

Dirk Lewandowski

Web search engines apply a variety of ranking signals to achieve user satisfaction, i.e., results pages that provide the best-possible results to the user. While these ranking signals implicitly consider credibility (e.g., by measuring popularity), explicit measures of credibility are not applied. In this chapter, credibility in Web search engines is discussed in a broad context: credibility as a measure for including documents in a search engine's index, credibility as a ranking signal, credibility in the context of universal search results, and the possibility of using credibility as an explicit measure for ranking purposes. It is found that while search engines-at least to a certain extent-show credible results to their users, there is no fully integrated credibility framework for Web search engines.

Georg Singer, Ulrich Norbisrath, Dirk Lewandowski

Search engines are the preferred tools for finding information on the Web. They are advancing to be the common helpers to answer any of our search needs. We use them to carry out simple look-up tasks and also to work on rather time consuming and more complex search tasks. Yet, we do not know very much about the user performance while carrying out those tasks -- especially not for ordinary users. The aim of this study was to get more insight into whether Web users manage to assess difficulty, time effort, query effort, and task outcome of search tasks, and if their judging performance relates to task complexity. Our study was conducted with a systematically selected sample of 56 people with a wide demographic background. They carried out a set of 12 search tasks with commercial Web search engines in a laboratory environment. The results confirm that it is hard for normal Web users to judge the difficulty and effort to carry out complex search tasks. The judgments are more reliable for simple tasks than for complex ones. Task complexity is an indicator for judging performance.

Georg Singer, Ulrich Norbisrath, Dirk Lewandowski

More and more people use the Internet to work on duties of their daily work routine. To find the right information online, Web search engines are the tools of their choice. Apart from finding facts, people use Web search engines to also execute rather complex and time consuming search tasks. So far search engines follow the one-for-all approach to serve its users and little is known about the impact of gender and age on people's Web search behavior. In this article we present a study that examines (1) how female and male web users carry out simple and complex search tasks and what are the differences between the two user groups, and (2) how the age of the users impacts their search performance. The laboratory study was done with 56 ordinary people each carrying out 12 search tasks. Our findings confirm that age impacts behavior and search performance significantly, while gender influences were smaller than expected.

Georg Singer, Ulrich Norbisrath, Dirk Lewandowski

Web search engines have become the dominant tools for finding information on the Internet. Due to their popularity, users apply them to a wide range of search needs, from simple look-ups to rather complex information tasks. This paper presents the results of a study to investigate the characteristics of these complex information needs in the context of Web search engines. The aim of the study is to find out more about (1) what makes complex search tasks distinct from simple tasks and if it is possible to find simple measures for describing their complexity, (2) if search success for a task can be predicted by means of unique measures, and (3) if successful searchers show a different behavior than unsuccessful ones. The study includes 60 people who carried out a set of 12 search tasks with current commercial search engines. Their behavior was logged with the Search-Logger tool. The results confirm that complex tasks show significantly different characteristics than simple tasks. Yet it seems to be difficult to distinguish successful from unsuccessful search behaviors. Good searchers can be differentiated from bad searchers by means of measurable parameters. The implications of these findings for search engine vendors are discussed.