PolyViT: Co-training Vision Transformers on Images, Videos and Audio
This work addresses the challenge of parameter-efficient and generalizable multi-modal learning for AI systems, though it is incremental in combining existing co-training methods.
The authors tackled the problem of training a single transformer model to process multiple modalities (images, videos, audio) and datasets with shared parameters, achieving state-of-the-art results on 5 video- and audio-classification datasets.
Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters? We present PolyViT, a model trained on image, audio and video which answers this question. By co-training different tasks on a single modality, we are able to improve the accuracy of each individual task and achieve state-of-the-art results on 5 standard video- and audio-classification datasets. Co-training PolyViT on multiple modalities and tasks leads to a model that is even more parameter-efficient, and learns representations that generalize across multiple domains. Moreover, we show that co-training is simple and practical to implement, as we do not need to tune hyperparameters for each combination of datasets, but can simply adapt those from standard, single-task training.