Automatic Generation of Web Censorship Probe Lists
This work addresses the scalability and accuracy issues in web censorship measurement for researchers and organizations monitoring internet freedom, representing a novel method for automating a previously manual process.
The paper tackled the problem of generating comprehensive and up-to-date domain probe lists for web censorship measurement by developing an automated method that analyzes existing URLs, extracts topics, and uses search engines to discover new candidate pages. The result was the generation of 119,255 new URLs across 35,147 domains, with measurements revealing over 1,400 domains suspected to be blocked that were not in the original dataset.
Domain probe lists--used to determine which URLs to probe for Web censorship--play a critical role in Internet censorship measurement studies. Indeed, the size and accuracy of the domain probe list limits the set of censored pages that can be detected; inaccurate lists can lead to an incomplete view of the censorship landscape or biased results. Previous efforts to generate domain probe lists have been mostly manual or crowdsourced. This approach is time-consuming, prone to errors, and does not scale well to the ever-changing censorship landscape. In this paper, we explore methods for automatically generating probe lists that are both comprehensive and up-to-date for Web censorship measurement. We start from an initial set of 139,957 unique URLs from various existing test lists consisting of pages from a variety of languages to generate new candidate pages. By analyzing content from these URLs (i.e., performing topic and keyword extraction), expanding these topics, and using them as a feed to search engines, our method produces 119,255 new URLs across 35,147 domains. We then test the new candidate pages by attempting to access each URL from servers in eleven different global locations over a span of four months to check for their connectivity and potential signs of censorship. Our measurements reveal that our method discovered over 1,400 domains--not present in the original dataset--we suspect to be blocked. In short, automatically updating probe lists is possible, and can help further automate censorship measurements at scale.