PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics
This work addresses the need for physics researchers to have language models that can generate coherent explanations, though it is incremental as it focuses on evaluation rather than new model development.
The authors tackled the problem of evaluating language models' ability to produce coherent explanations in physics by creating a dataset collection that measures sentence ordering, position, and discourse coherence, revealing that current models struggle with these tasks even when trained on mathematical language objectives.
In order for language models to aid physics research, they must first encode representations of mathematical and natural language discourse which lead to coherent explanations, with correct ordering and relevance of statements. We present a collection of datasets developed to evaluate the performance of language models in this regard, which measure capabilities with respect to sentence ordering, position, section prediction, and discourse coherence. Analysis of the data reveals equations and sub-disciplines which are most common in physics discourse, as well as the sentence-level frequency of equations and expressions. We present baselines that demonstrate how contemporary language models are challenged by coherence related tasks in physics, even when trained on mathematical natural language objectives.