Compliance Rating Scheme: A Data Provenance Framework for Generative AI Datasets
This addresses ethical and legal issues in dataset creation for AI researchers and practitioners, though it is incremental as it builds on existing data provenance technology.
The paper tackles the problem of opaque and unethical data collection practices in generative AI datasets by introducing the Compliance Rating Scheme (CRS), a framework to evaluate dataset compliance with transparency, accountability, and security principles, and releases an open-source Python library for implementation.
Generative Artificial Intelligence (GAI) has experienced exponential growth in recent years, partly facilitated by the abundance of large-scale open-source datasets. These datasets are often built using unrestricted and opaque data collection practices. While most literature focuses on the development and applications of GAI models, the ethical and legal considerations surrounding the creation of these datasets are often neglected. In addition, as datasets are shared, edited, and further reproduced online, information about their origin, legitimacy, and safety often gets lost. To address this gap, we introduce the Compliance Rating Scheme (CRS), a framework designed to evaluate dataset compliance with critical transparency, accountability, and security principles. We also release an open-source Python library built around data provenance technology to implement this framework, allowing for seamless integration into existing dataset-processing and AI training pipelines. The library is simultaneously reactive and proactive, as in addition to evaluating the CRS of existing datasets, it equally informs responsible scraping and construction of new datasets.