A Review on Oracle Issues in Machine Learning
It addresses data reliability problems for ML practitioners, but is incremental as a review paper.
The paper surveys oracle issues in machine learning, where data may not accurately represent the problem, and reviews state-of-the-art solutions including differential testing, metamorphic testing, and tools for dataset-specific issues.
Machine learning contrasts with traditional software development in that the oracle is the data, and the data is not always a correct representation of the problem that machine learning tries to model. We present a survey of the oracle issues found in machine learning and state-of-the-art solutions for dealing with these issues. These include lines of research for differential testing, metamorphic testing, and test coverage. We also review some recent improvements to robustness during modeling that reduce the impact of oracle issues, as well as tools and frameworks for assisting in testing and discovering issues specific to the dataset.