Adversarial Learning for Feature Shift Detection and Correction
This addresses data shift issues in real-world applications like biomedical and financial data, offering a novel approach for feature-level correction, though it builds on existing adversarial learning concepts.
The paper tackles the problem of detecting and correcting feature shifts in datasets, such as those from malfunctioning sensors or faulty data processing, by using adversarial learning principles with discriminators to localize and corrupt features, resulting in outperformance of existing statistical and neural network-based methods.
Data shift is a phenomenon present in many real-world applications, and while there are multiple methods attempting to detect shifts, the task of localizing and correcting the features originating such shifts has not been studied in depth. Feature shifts can occur in many datasets, including in multi-sensor data, where some sensors are malfunctioning, or in tabular and structured data, including biomedical, financial, and survey data, where faulty standardization and data processing pipelines can lead to erroneous features. In this work, we explore using the principles of adversarial learning, where the information from several discriminators trained to distinguish between two distributions is used to both detect the corrupted features and fix them in order to remove the distribution shift between datasets. We show that mainstream supervised classifiers, such as random forest or gradient boosting trees, combined with simple iterative heuristics, can localize and correct feature shifts, outperforming current statistical and neural network-based techniques. The code is available at https://github.com/AI-sandbox/DataFix.