Fast Line Search for Multi-Task Learning
This work addresses optimization inefficiencies in multi-task learning, which is incremental as it builds on existing line search techniques.
The authors tackled the problem of slow line search in multi-task learning by proposing a method that uses latent representation space instead of parameter space to find step sizes, resulting in more accurate and faster solutions compared to traditional backtracking while maintaining competitive computational time and performance against constant learning rate methods.
Multi-task learning is a powerful method for solving several tasks jointly by learning robust representation. Optimization of the multi-task learning model is a more complex task than a single-task due to task conflict. Based on theoretical results, convergence to the optimal point is guaranteed when step size is chosen through line search. But, usually, line search for the step size is not the best choice due to the large computational time overhead. We propose a novel idea for line search algorithms in multi-task learning. The idea is to use latent representation space instead of parameter space for finding step size. We examined this idea with backtracking line search. We compare this fast backtracking algorithm with classical backtracking and gradient methods with a constant learning rate on MNIST, CIFAR-10, Cityscapes tasks. The systematic empirical study showed that the proposed method leads to more accurate and fast solution, than the traditional backtracking approach and keep competitive computational time and performance compared to the constant learning rate method.