TCT: A Cross-supervised Learning Method for Multimodal Sequence Representation
This work addresses the problem of multimodal representation learning for researchers in AI and NLP, though it appears incremental as it builds on existing Transformer methods.
The authors tackled the challenge of efficiently learning semantic representations from multimodal sequences by proposing TCT, a Transformer-based cross-modal translator, which when combined with MTN achieved new state-of-the-art performance on video-grounded dialogue tasks.
Multimodalities provide promising performance than unimodality in most tasks. However, learning the semantic of the representations from multimodalities efficiently is extremely challenging. To tackle this, we propose the Transformer based Cross-modal Translator (TCT) to learn unimodal sequence representations by translating from other related multimodal sequences on a supervised learning method. Combined TCT with Multimodal Transformer Network (MTN), we evaluate MTN-TCT on the video-grounded dialogue which uses multimodality. The proposed method reports new state-of-the-art performance on video-grounded dialogue which indicates representations learned by TCT are more semantics compared to directly use unimodality.