ATE-SG: Alternate Through the Epochs Stochastic Gradient for Multi-Task Neural Networks
This addresses training challenges in multi-task neural networks, though it appears incremental.
The paper tackles the problem of conflicting loss gradients in hard-parameter sharing multi-task neural networks by proposing an alternate training method that updates shared and task-specific weights alternately through epochs, resulting in enhanced training regularization and reduced computational demands.
This paper introduces novel alternate training procedures for hard-parameter sharing Multi-Task Neural Networks (MTNNs). Traditional MTNN training faces challenges in managing conflicting loss gradients, often yielding sub-optimal performance. The proposed alternate training method updates shared and task-specific weights alternately through the epochs, exploiting the multi-head architecture of the model. This approach reduces computational costs per epoch and memory requirements. Convergence properties similar to those of the classical stochastic gradient method are established. Empirical experiments demonstrate enhanced training regularization and reduced computational demands. In summary, our alternate training procedures offer a promising advancement for the training of hard-parameter sharing MTNNs.