Mining Associated Text and Images with Dual-Wing Harmoniums
This work addresses the challenge of analyzing associated text and images for applications like news video collections, presenting an incremental improvement over prior models.
The authors tackled the problem of mining multimedia data by proposing a dual-wing harmonium model that extends earlier two-layer random fields, offering efficient inference and robust topic mixing. They applied the model to captioned images using multivariate Poisson for word-counts and Gaussian for color histograms, and reported empirical results on classification, retrieval, and image annotation tasks with comparisons to existing models.
We propose a multi-wing harmonium model for mining multimedia data that extends and improves on earlier models based on two-layer random fields, which capture bidirectional dependencies between hidden topic aspects and observed inputs. This model can be viewed as an undirected counterpart of the two-layer directed models such as LDA for similar tasks, but bears significant difference in inference/learning cost tradeoffs, latent topic representations, and topic mixing mechanisms. In particular, our model facilitates efficient inference and robust topic mixing, and potentially provides high flexibilities in modeling the latent topic spaces. A contrastive divergence and a variational algorithm are derived for learning. We specialized our model to a dual-wing harmonium for captioned images, incorporating a multivariate Poisson for word-counts and a multivariate Gaussian for color histogram. We present empirical results on the applications of this model to classification, retrieval and image annotation on news video collections, and we report an extensive comparison with various extant models.