Characterizing Web Search in The Age of Generative AI
This work addresses the evolving nature of web search for users and researchers in the age of Generative AI, though it is incremental as it characterizes differences rather than proposing new methods.
The paper compares generative search engines, which use LLMs to produce coherent text responses from retrieved web pages, with traditional web search engines like Google, analyzing differences across queries from four domains. It finds that generative search engines cover a wider range of sources, vary in reliance on internal vs. external knowledge, and surface diverse concepts, highlighting the need for updated evaluation criteria.
The advent of LLMs has given rise to a new type of web search: Generative search, where LLMs retrieve web pages related to a query and generate a single, coherent text as a response. This output modality stands in stark contrast to traditional web search, where results are returned as a ranked list of independent web pages. In this paper, we ask: Along what dimensions do generative search outputs differ from traditional web search? We compare Google, a traditional web search engine, with four generative search engines from two providers (Google and OpenAI) across queries from four domains. Our analysis reveals intriguing differences. Most generative search engines cover a wider range of sources compared to web search. Generative search engines vary in the degree to which they rely on internal knowledge contained within the model parameters v.s. external knowledge retrieved from the web. Generative search engines surface varying sets of concepts, creating new opportunities for enhancing search diversity and serendipity. Our results also highlight the need for revisiting evaluation criteria for web search in the age of Generative AI.