DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems
This addresses the problem of unreliable ML deployments for practitioners and researchers by providing a structured approach, though it is incremental as it builds on the emerging data-centric AI paradigm without introducing new methods.
The authors tackled the lack of standardized guidance for data-centric AI in machine learning systems by proposing DC-Check, a checklist framework to elicit data-centric considerations across the ML pipeline, resulting in an actionable tool aimed at improving reliability and transparency.
While there have been a number of remarkable breakthroughs in machine learning (ML), much of the focus has been placed on model development. However, to truly realize the potential of machine learning in real-world settings, additional aspects must be considered across the ML pipeline. Data-centric AI is emerging as a unifying paradigm that could enable such reliable end-to-end pipelines. However, this remains a nascent area with no standardized framework to guide practitioners to the necessary data-centric considerations or to communicate the design of data-centric driven ML systems. To address this gap, we propose DC-Check, an actionable checklist-style framework to elicit data-centric considerations at different stages of the ML pipeline: Data, Training, Testing, and Deployment. This data-centric lens on development aims to promote thoughtfulness and transparency prior to system development. Additionally, we highlight specific data-centric AI challenges and research opportunities. DC-Check is aimed at both practitioners and researchers to guide day-to-day development. As such, to easily engage with and use DC-Check and associated resources, we provide a DC-Check companion website (https://www.vanderschaar-lab.com/dc-check/). The website will also serve as an updated resource as methods and tooling evolve over time.