Large-scale text processing pipeline with Apache Spark
This enables political science researchers to study policy diffusion more comprehensively, though it's an incremental application of existing tools to a new domain.
The authors tackled the computational bottleneck preventing all-pairs comparison of bills for policy diffusion detection across U.S. state legislatures by implementing a distributed text processing pipeline with Apache Spark, enabling analysis of the full dataset instead of single topics.
In this paper, we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas. We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark dataframes and Scala application programming interface. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.