FastWARC: Optimizing Large-Scale Web Archive Analytics
This work addresses a bottleneck in web data analytics for researchers and engineers handling terabyte- or petabyte-scale archives, though it is incremental as it optimizes an existing standard format.
The paper tackled the inefficiency of processing large-scale web archives by developing FastWARC, a high-performance WARC processing library for Python, which achieved performance improvements of 1.6-8x over existing tools.
Web search and other large-scale web data analytics rely on processing archives of web pages stored in a standardized and efficient format. Since its introduction in 2008, the IIPC's Web ARCive (WARC) format has become the standard format for this purpose. As a list of individually compressed records of HTTP requests and responses, it allows for constant-time random access to all kinds of web data via off-the-shelf open source parsers in many programming languages, such as WARCIO, the de-facto standard for Python. When processing web archives at the terabyte or petabyte scale, however, even small inefficiencies in these tools add up quickly, resulting in hours, days, or even weeks of wasted compute time. Reviewing the basic components of WARCIO and analyzing its bottlenecks, we proceed to build FastWARC, a new high-performance WARC processing library for Python, written in C++/Cython, which yields performance improvements by factors of 1.6-8x.