Polynomial-based Self-Attention for Table Representation learning
This addresses a key bottleneck in table representation learning for machine learning applications, though it is incremental as it modifies an existing component.
The paper tackles the oversmoothing issue in Transformers for tabular data by proposing a novel matrix polynomial-based self-attention layer, which enhances scalability and outperforms state-of-the-art methods in representation performance.
Structured data, which constitutes a significant portion of existing data types, has been a long-standing research topic in the field of machine learning. Various representation learning methods for tabular data have been proposed, ranging from encoder-decoder structures to Transformers. Among these, Transformer-based methods have achieved state-of-the-art performance not only in tabular data but also in various other fields, including computer vision and natural language processing. However, recent studies have revealed that self-attention, a key component of Transformers, can lead to an oversmoothing issue. We show that Transformers for tabular data also face this problem, and to address the problem, we propose a novel matrix polynomial-based self-attention layer as a substitute for the original self-attention layer, which enhances model scalability. In our experiments with three representative table learning models equipped with our proposed layer, we illustrate that the layer effectively mitigates the oversmoothing problem and enhances the representation performance of the existing methods, outperforming the state-of-the-art table representation methods.