Multi-branch of Attention Yields Accurate Results for Tabular Data
This addresses the challenge of handling heterogeneous features in tabular data for machine learning practitioners, though it appears incremental as it builds on existing transformer methods.
The paper tackles the problem of feature heterogeneity in tabular data by proposing MAYA, a transformer-based framework with a Multi-Branch of Attention encoder and collaborative learning, which achieves superior performance in classification and regression tasks compared to other transformer-based methods.
Tabular data inherently exhibits significant feature heterogeneity, but existing transformer-based methods lack specialized mechanisms to handle this property. To bridge the gap, we propose MAYA, an encoder-decoder transformer-based framework. In the encoder, we design a Multi-Branch of Attention (MBA) that constructs multiple parallel attention branches and averages the features at each branch, effectively fusing heterogeneous features while limiting parameter growth. Additionally, we employ collaborative learning with a dynamic consistency weight constraint to produce more robust representations. In the decoder stage, cross-attention is utilized to seamlessly integrate tabular data with corresponding label features. This dual-attention mechanism effectively captures both intra-instance and inter-instance interactions. We evaluate the proposed method on a wide range of datasets and compare it with other state-of-the-art transformer-based methods. Extensive experiments demonstrate that our model achieves superior performance among transformer-based methods in both tabular classification and regression tasks.