Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?
This addresses the problem of task conflict in multi-task learning for NLP researchers, providing insights into the robustness of text-to-text models, but it is incremental as it builds on existing multi-task learning studies.
The paper investigates whether text-to-text multi-task learning architectures, which use a shared encoder and language model decoder, experience task conflicts that require specialized training algorithms, finding that both directional conflict and transfer remain surprisingly constant across architectures.
Traditional multi-task learning architectures train a single model across multiple tasks through a shared encoder followed by task-specific decoders. Learning these models often requires specialized training algorithms that address task-conflict in the shared parameter updates, which otherwise can lead to negative transfer. A new type of multi-task learning within NLP homogenizes multi-task architectures as a shared encoder and language model decoder, which does surprisingly well across a range of diverse tasks. Does this new architecture suffer from task-conflicts that require specialized training algorithms? We study how certain factors in the shift towards text-to-text models affects multi-task conflict and negative transfer, finding that both directional conflict and transfer are surprisingly constant across architectures.