LakeHopper: Cross Data Lakes Column Type Annotation through Model Adaptation
This addresses the problem of reducing annotation effort for data lake users in tasks like data cleaning and integration, but it is incremental as it builds on existing LM-based methods.
The paper tackles the problem of adapting a pre-trained language model for column type annotation from a source data lake to a target data lake with minimal new annotations, proposing LakeHopper, which achieved effective results in experiments on two data lake transfers under low- and high-resource settings.
Column type annotation is vital for tasks like data cleaning, integration, and visualization. Recent solutions rely on resource-intensive language models fine-tuned on well-annotated columns from a particular set of tables, i.e., a source data lake. In this paper, we study whether we can adapt an existing pre-trained LM-based model to a new (i.e., target) data lake to minimize the annotations required on the new data lake. However, challenges include the source-target knowledge gap, selecting informative target data, and fine-tuning without losing shared knowledge exist. We propose LakeHopper, a framework that identifies and resolves the knowledge gap through LM interactions, employs a cluster-based data selection scheme for unannotated columns, and uses an incremental fine-tuning mechanism that gradually adapts the source model to the target data lake. Our experimental results validate the effectiveness of LakeHopper on two different data lake transfers under both low-resource and high-resource settings.