Tabular Data Adapters: Improving Outlier Detection for Unlabeled Private Data
This addresses the cold start problem in labeling for outlier detection in industrial applications, though it is incremental as it builds on existing public models.
The paper tackles the challenge of applying deep learning to private tabular data by introducing Tabular Data Adapters (TDA), a method that generates soft labels for outlier detection using public datasets, resulting in more accurate annotations and reduced computational time across 50 datasets.
The remarkable success of Deep Learning approaches is often based and demonstrated on large public datasets. However, when applying such approaches to internal, private datasets, one frequently faces challenges arising from structural differences in the datasets, domain shift, and the lack of labels. In this work, we introduce Tabular Data Adapters (TDA), a novel method for generating soft labels for unlabeled tabular data in outlier detection tasks. By identifying statistically similar public datasets and transforming private data (based on a shared autoencoder) into a format compatible with state-of-the-art public models, our approach enables the generation of weak labels. It thereby can help to mitigate the cold start problem of labeling by basing on existing outlier detection models for public datasets. In experiments on 50 tabular datasets across different domains, we demonstrate that our method is able to provide more accurate annotations than baseline approaches while reducing computational time. Our approach offers a scalable, efficient, and cost-effective solution, to bridge the gap between public research models and real-world industrial applications.