Technical report: Training Mixture Density Networks with full covariance matrices
This technical report provides a documented implementation for researchers working on conditional probability modeling problems where covariance between variables is important, though it appears to be an incremental contribution.
The authors tackled the limitation of standard Mixture Density Networks having restricted covariance matrices by deriving and implementing an MDN formulation with unrestricted covariances, though they note this approach has likely been done before but wasn't documented online.
Mixture Density Networks are a tried and tested tool for modelling conditional probability distributions. As such, they constitute a great baseline for novel approaches to this problem. In the standard formulation, an MDN takes some input and outputs parameters for a Gaussian mixture model with restrictions on the mixture components' covariance. Since covariance between random variables is a central issue in the conditional modeling problems we were investigating, I derived and implemented an MDN formulation with unrestricted covariances. It is likely that this has been done before, but I could not find any resources online. For this reason, I have documented my approach in the form of this technical report, in hopes that it may be useful to others facing a similar situation.