Efficient Computation Sharing for Multi-Task Visual Scene Understanding
This work addresses efficiency challenges in multi-task learning for visual scene understanding, offering a solution that reduces computational and storage costs while maintaining performance, though it is incremental as it builds on existing transfer learning and multi-task techniques.
The paper tackles the problem of resource-intensive multi-task visual scene understanding by proposing a computation- and parameter-sharing framework that balances efficiency and accuracy, achieving higher accuracy and reduced computational resources compared to state-of-the-art transformer-based methods on NYUD-v2 and PASCAL-context datasets.
Solving multiple visual tasks using individual models can be resource-intensive, while multi-task learning can conserve resources by sharing knowledge across different tasks. Despite the benefits of multi-task learning, such techniques can struggle with balancing the loss for each task, leading to potential performance degradation. We present a novel computation- and parameter-sharing framework that balances efficiency and accuracy to perform multiple visual tasks utilizing individually-trained single-task transformers. Our method is motivated by transfer learning schemes to reduce computational and parameter storage costs while maintaining the desired performance. Our approach involves splitting the tasks into a base task and the other sub-tasks, and sharing a significant portion of activations and parameters/weights between the base and sub-tasks to decrease inter-task redundancies and enhance knowledge sharing. The evaluation conducted on NYUD-v2 and PASCAL-context datasets shows that our method is superior to the state-of-the-art transformer-based multi-task learning techniques with higher accuracy and reduced computational resources. Moreover, our method is extended to video stream inputs, further reducing computational costs by efficiently sharing information across the temporal domain as well as the task domain. Our codes and models will be publicly available.