Multidimensional counting grids: Inferring word order from disordered bags of words
This work addresses the challenge of inferring latent order from disordered data for researchers in fields like biology, text analysis, and computer vision, representing a novel method for a known bottleneck.
The authors tackled the problem of modeling bag-of-words data that exhibits smooth thematic shifts across documents, rather than topic mixing, by extending window-based strategies to multiple dimensions and cases without obvious ordering. They demonstrated that this approach outperforms standard topic models in classification and prediction tasks across biology, text modeling, and computer vision applications, though no concrete numbers are provided.
Models of bags of words typically assume topic mixing so that the words in a single bag come from a limited number of topics. We show here that many sets of bag of words exhibit a very different pattern of variation than the patterns that are efficiently captured by topic mixing. In many cases, from one bag of words to the next, the words disappear and new ones appear as if the theme slowly and smoothly shifted across documents (providing that the documents are somehow ordered). Examples of latent structure that describe such ordering are easily imagined. For example, the advancement of the date of the news stories is reflected in a smooth change over the theme of the day as certain evolving news stories fall out of favor and new events create new stories. Overlaps among the stories of consecutive days can be modeled by using windows over linearly arranged tight distributions over words. We show here that such strategy can be extended to multiple dimensions and cases where the ordering of data is not readily obvious. We demonstrate that this way of modeling covariation in word occurrences outperforms standard topic models in classification and prediction tasks in applications in biology, text modeling and computer vision.