Learning More May Not Be Better: Knowledge Transferability in Vision and Language Tasks
This work addresses the assumption in machine learning that more data leads to better performance, showing it can be incremental for multi-modal tasks.
The study investigates whether combining multiple datasets improves vision-and-language models, finding that knowledge transfer is not always beneficial and depends on factors like task grouping and dataset size.
Is more data always better to train vision-and-language models? We study knowledge transferability in multi-modal tasks. The current tendency in machine learning is to assume that by joining multiple datasets from different tasks their overall performance will improve. However, we show that not all the knowledge transfers well or has a positive impact on related tasks, even when they share a common goal. We conduct an exhaustive analysis based on hundreds of cross-experiments on 12 vision-and-language tasks categorized in 4 groups. Whereas tasks in the same group are prone to improve each other, results show that this is not always the case. Other factors such as dataset size or pre-training stage have also a great impact on how well the knowledge is transferred.