Leveraging Task Structures for Improved Identifiability in Neural Network Representations
It addresses identifiability issues in neural network representations for researchers in machine learning, offering incremental theoretical extensions with practical applications.
This work tackles the problem of identifiability in supervised learning by leveraging task distributions, showing that linear identifiability is achievable in multi-task regression and that a conditional prior reduces equivalence classes to permutations and scaling. Empirically, the model outperforms unsupervised models in recovering canonical representations for synthetic and real-world molecular data.
This work extends the theory of identifiability in supervised learning by considering the consequences of having access to a distribution of tasks. In such cases, we show that linear identifiability is achievable in the general multi-task regression setting. Furthermore, we show that the existence of a task distribution which defines a conditional prior over latent factors reduces the equivalence class for identifiability to permutations and scaling of the true latent factors, a stronger and more useful result than linear identifiability. Crucially, when we further assume a causal structure over these tasks, our approach enables simple maximum marginal likelihood optimization, and suggests potential downstream applications to causal representation learning. Empirically, we find that this straightforward optimization procedure enables our model to outperform more general unsupervised models in recovering canonical representations for both synthetic data and real-world molecular data.