Exploring Transformer Placement in Variational Autoencoders for Tabular Data Generation
This work addresses the problem of generating realistic tabular data for domains like healthcare or finance, but it is incremental as it focuses on architectural placement rather than a breakthrough.
The paper tackled the challenge of generating tabular data by exploring where to place Transformers within Variational Autoencoders to improve feature relationship modeling, finding that positioning them in latent and decoder components leads to a trade-off between fidelity and diversity.
Tabular data remains a challenging domain for generative models. In particular, the standard Variational Autoencoder (VAE) architecture, typically composed of multilayer perceptrons, struggles to model relationships between features, especially when handling mixed data types. In contrast, Transformers, through their attention mechanism, are better suited for capturing complex feature interactions. In this paper, we empirically investigate the impact of integrating Transformers into different components of a VAE. We conduct experiments on 57 datasets from the OpenML CC18 suite and draw two main conclusions. First, results indicate that positioning Transformers to leverage latent and decoder representations leads to a trade-off between fidelity and diversity. Second, we observe a high similarity between consecutive blocks of a Transformer in all components. In particular, in the decoder, the relationship between the input and output of a Transformer is approximately linear.