A Primer on the Data Cleaning Pipeline
It offers a primer for analysts and researchers dealing with data integration challenges across various domains, but it is incremental as it reviews existing methods rather than introducing new ones.
The paper addresses the growing need for data cleaning due to the rapid expansion of structured and unstructured databases, providing a review of the data cleaning pipeline with its four stages to enable downstream analyses on cleaned data.
The availability of both structured and unstructured databases, such as electronic health data, social media data, patent data, and surveys that are often updated in real time, among others, has grown rapidly over the past decade. With this expansion, the statistical and methodological questions around data integration, or rather merging multiple data sources, has also grown. Specifically, the science of the ``data cleaning pipeline'' contains four stages that allow an analyst to perform downstream tasks, predictive analyses, or statistical analyses on ``cleaned data.'' This article provides a review of this emerging field, introducing technical terminology and commonly used methods.