Improving Data Quality through Deep Learning and Statistical Models
This work addresses data quality issues for users in domains relying on accurate data, but it appears incremental as it combines existing deep learning and statistical methods without claiming major breakthroughs.
The authors tackled the problem of inefficient and inaccurate traditional data quality control by proposing a deep learning and statistical model framework, demonstrating its application on an open salary dataset to identify outliers and improve data quality.
Traditional data quality control methods are based on users experience or previously established business rules, and this limits performance in addition to being a very time consuming process with lower than desirable accuracy. Utilizing deep learning, we can leverage computing resources and advanced techniques to overcome these challenges and provide greater value to users. In this paper, we, the authors, first review relevant works and discuss machine learning techniques, tools, and statistical quality models. Second, we offer a creative data quality framework based on deep learning and statistical model algorithm for identifying data quality. Third, we use data involving salary levels from an open dataset published by the state of Arkansas to demonstrate how to identify outlier data and how to improve data quality via deep learning. Finally, we discuss future work.