Toward an AI-Native Internet: Rethinking the Web Architecture for Semantic Retrieval
This addresses inefficiencies in web architecture for AI applications like LLMs, though it appears incremental as it builds on existing web concepts.
The paper tackles the problem that current web architecture is inefficient for AI-driven semantic retrieval, proposing an AI-Native Internet where servers expose semantic information chunks instead of full documents, with experiments quantifying inefficiencies in HTML-based retrieval.
The rise of Generative AI Search is fundamentally transforming how users and intelligent systems interact with the Internet. LLMs increasingly act as intermediaries between humans and web information. Yet the web remains optimized for human browsing rather than AI-driven semantic retrieval, resulting in wasted network bandwidth, lower information quality, and unnecessary complexity for developers. We introduce the concept of an AI-Native Internet, a web architecture in which servers expose semantically relevant information chunks rather than full documents, supported by a Web-native semantic resolver that allows AI applications to discover relevant information sources before retrieving fine-grained chunks. Through motivational experiments, we quantify the inefficiencies of current HTML-based retrieval, and outline architectural directions and open challenges for evolving today's document-centric web into an AI-oriented substrate that better supports semantic access to web content.