Tabby: Tabular Data Synthesis with Language Models
This addresses the need for high-quality synthetic tabular data for applications like data augmentation and privacy preservation, though it is incremental as it builds on existing language model architectures.
The paper tackles the problem of synthesizing tabular data, which has received less attention compared to text data, by introducing Tabby, a post-training modification to Transformer language models that uses Gated Mixture-of-Experts for column-specific representations. The result includes up to a 44% improvement in data quality over previous methods and achieves near or equal quality to real data on tabular and nested JSON datasets.
While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.