The Importance of Encoder Choice:A Tabular-Image Study
For researchers in multimodal learning, this work demonstrates that the choice of tabular encoder significantly impacts performance, challenging the common practice of using a plain MLP.
This study evaluates state-of-the-art tabular models as encoders in image-tabular multimodal learning, finding that using a modern tabular encoder (e.g., TabPFN) improves performance by up to 5% over a plain MLP encoder, highlighting the importance of encoder choice.
Multimodal learning usually requires a dedicated encoder per modality. When a tabular modality is involved, prior work has been mostly using a \emph{plain MLP} as the encoder. Yet if it were a strong encoder, the tabular domain would not be ``the last unconquered castle for deep learning''. This study evaluates state-of-the-art tabular models as encoders in the image-tabular setting for the first time. An obstacle stands out. In-Context Learning models, among the best performing methods in the tabular domain, require labels to process instances, making it non-trivial to embed training and test instances the same way. We addressed this problem across multiple models of this family. With this study, we would like to highlight the importance of encoder factor in the multimodal learning.