TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
This addresses the need for models that can handle both free-form text and structured tabular data, which is crucial for tasks like semantic parsing, though it is incremental as it builds on existing pretrained language model approaches.
The authors tackled the problem of semantic parsing over structured data by introducing TaBERT, a pretrained language model that jointly learns representations for natural language sentences and (semi-)structured tables, achieving new best results on WikiTableQuestions and competitive performance on Spider.
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .