iCrawl: Improving the Freshness of Web Collections by Integrating Social Web and Focused Web Crawling
This addresses the need for researchers and journalists to monitor fresh online content, though it is incremental as it builds on existing focused crawling methods.
The paper tackles the problem of collecting fresh and thematically coherent web content for current events by integrating social media streams into focused crawling, resulting in a novel crawler that improves freshness.
Researchers in the Digital Humanities and journalists need to monitor, collect and analyze fresh online content regarding current events such as the Ebola outbreak or the Ukraine crisis on demand. However, existing focused crawling approaches only consider topical aspects while ignoring temporal aspects and therefore cannot achieve thematically coherent and fresh Web collections. Especially Social Media provide a rich source of fresh content, which is not used by state-of-the-art focused crawlers. In this paper we address the issues of enabling the collection of fresh and relevant Web and Social Web content for a topic of interest through seamless integration of Web and Social Media in a novel integrated focused crawler. The crawler collects Web and Social Media content in a single system and exploits the stream of fresh Social Media content for guiding the crawler.