BUbiNG: Massive Crawling for the Masses
This provides a scalable, high-performance web crawling solution for researchers and developers, addressing a gap in open-source tools.
The authors tackled the lack of freely available, open-source web crawlers that guarantee high throughput and linear scalability by developing BUbiNG, a fully distributed Java crawler that can crawl several thousand pages per second while respecting politeness constraints.
Although web crawlers have been around for twenty years by now, there is virtually no freely available, opensource crawling software that guarantees high throughput, overcomes the limits of single-machine systems and at the same time scales linearly with the amount of resources available. This paper aims at filling this gap, through the description of BUbiNG, our next-generation web crawler built upon the authors' experience with UbiCrawler [Boldi et al. 2004] and on the last ten years of research on the topic. BUbiNG is an opensource Java fully distributed crawler; a single BUbiNG agent, using sizeable hardware, can crawl several thousands pages per second respecting strict politeness constraints, both host- and IP-based. Unlike existing open-source distributed crawlers that rely on batch techniques (like MapReduce), BUbiNG job distribution is based on modern high-speed protocols so to achieve very high throughput.