Design and Implementation of Domain based Semantic Hidden Web Crawler
This addresses the challenge of accessing large amounts of hidden web data for users needing domain-specific information, though it appears incremental as it builds on existing semantic methods.
The paper tackles the problem of extracting data from hidden web pages behind HTML search forms, which traditional crawlers cannot access, by proposing a technique that uses semantic mapping with domain-specific databases to fill forms, resulting in more accurate and qualitative data extraction.
Web is a wide term which mainly consists of surface web and hidden web. One can easily access the surface web using traditional web crawlers, but they are not able to crawl the hidden portion of the web. These traditional crawlers retrieve contents from web pages, which are linked by hyperlinks ignoring the information hidden behind form pages, which cannot be extracted using simple hyperlink structure. Thus, they ignore large amount of data hidden behind search forms. This paper emphasizes on the extraction of hidden data behind html search forms. The proposed technique makes use of semantic mapping to fill the html search form using domain specific database. Using semantics to fill various fields of a form leads to more accurate and qualitative data extraction.