World of Code: Enabling a Research Workflow for Mining and Analyzing the Universe of Open Source VCS data
This work addresses the need for comprehensive data infrastructure to study global properties of open source software development, potentially increasing ecosystem resiliency, though it is incremental in building on existing data collection efforts.
The authors tackled the problem of limited understanding of the entire open source software ecosystem by creating World of Code (WoC), a large, frequently updated collection of version control data with over 18B Git objects, enabling cross-referencing and analysis to support research on trends, ecosystem measurement, and package usage.
Open source software (OSS) is essential for modern society and, while substantial research has been done on individual (typically central) projects, only a limited understanding of the periphery of the entire OSS ecosystem exists. For example, how are the tens of millions of projects in the periphery interconnected through. technical dependencies, code sharing, or knowledge flow? To answer such questions we: a) create a very large and frequently updated collection of version control data in the entire FLOSS ecosystems named World of Code (WoC), that can completely cross-reference authors, projects, commits, blobs, dependencies, and history of the FLOSS ecosystems and b) provide capabilities to efficiently correct, augment, query, and analyze that data. Our current WoC implementation is capable of being updated on a monthly basis and contains over 18B Git objects. To evaluate its research potential and to create vignettes for its usage, we employ WoC in conducting several research tasks. In particular, we find that it is capable of supporting trend evaluation, ecosystem measurement, and the determination of package usage. We expect WoC to spur investigation into global properties of OSS development leading to increased resiliency of the entire OSS ecosystem. Our infrastructure facilitates the discovery of key technical dependencies, code flow, and social networks that provide the basis to determine the structure and evolution of the relationships that drive FLOSS activities and innovation.