Why we need an independent index of the Web
This addresses the problem of lack of diversity in web search for users and developers, but is incremental as it focuses on enabling conditions rather than a direct solution.
The paper argues that building a proprietary index is a major barrier to creating diverse search engines, and proposes that overcoming this obstacle could enable the conditions for greater diversity in the search engine marketplace.
The path to greater diversity, as we have seen, cannot be achieved by merely hoping for a new search engine nor will government support for a single alternative achieve this goal. What is instead required is to create the conditions that will make establishing such a search engine possible in the first place. I describe how building and maintaining a proprietary index is the greatest deterrent to such an undertaking. We must first overcome this obstacle. Doing so will still not solve the problem of the lack of diversity in the search engine marketplace. But it may establish the conditions necessary to achieve that desired end.