Compressed Hierarchical Representations for Multi-Task Learning and Task Clustering
This work addresses multi-task learning efficiency and interpretability for machine learning practitioners, though it is incremental as it builds on existing hierarchical and information bottleneck methods.
The paper tackles homogeneous-feature multi-task learning by framing it as a hierarchical representation learning problem with task-agnostic and task-specific latents, using an information bottleneck and additive noise model to limit task-specific information, resulting in competitive performance on MTL benchmarks and parameters that relate to task similarity for interpretability.
In this paper, we frame homogeneous-feature multi-task learning (MTL) as a hierarchical representation learning problem, with one task-agnostic and multiple task-specific latent representations. Drawing inspiration from the information bottleneck principle and assuming an additive independent noise model between the task-agnostic and task-specific latent representations, we limit the information contained in each task-specific representation. It is shown that our resulting representations yield competitive performance for several MTL benchmarks. Furthermore, for certain setups, we show that the trained parameters of the additive noise model are closely related to the similarity of different tasks. This indicates that our approach yields a task-agnostic representation that is disentangled in the sense that its individual dimensions may be interpretable from a task-specific perspective.