Exploratory Analysis of a Terabyte Scale Web Corpus
This work provides incremental insights into web data analysis for researchers and practitioners, focusing on a specific dataset without broad methodological advances.
The authors tackled the problem of analyzing the Common Crawl Corpus, a large web dataset, by measuring nine web characteristics at two granularity levels using MapReduce, and they reported initial observations on a fraction of it, noting that language distribution and HTML version analysis were novel contributions.
In this paper we present a preliminary analysis over the largest publicly accessible web dataset: the Common Crawl Corpus. We measure nine web characteristics from two levels of granularity using MapReduce and we comment on the initial observations over a fraction of it. To the best of our knowledge two of the characteristics, the language distribution and the HTML version of pages have not been analyzed in previous work, while the specific dataset has been only analyzed on page level.