Context is Key: New Approaches to Neural Coherence Modeling
This work addresses coherence modeling in natural language processing, offering simpler and more efficient models that achieve competitive results, though it appears incremental by building on existing pairwise approaches.
The paper tackled the problem of neural coherence modeling by formulating it as a regression task and proposing two novel methods, including a context-adding technique that matches or exceeds state-of-the-art scores on Kendall-tau distance and positional accuracy metrics.
We formulate coherence modeling as a regression task and propose two novel methods to combine techniques from our setup with pairwise approaches. The first of our methods is a model that we call "first-next," which operates similarly to selection sorting but conditions decision-making on information about already-sorted sentences. The second consists of a technique for adding context to regression-based models by concatenating sentence-level representations with an encoding of its corresponding out-of-order paragraph. This latter model achieves Kendall-tau distance and positional accuracy scores that match or exceed the current state-of-the-art on these metrics. Our results suggest that many of the gains that come from more complex, machine-translation inspired approaches can be achieved with simpler, more efficient models.