Methods for Estimating and Improving Robustness of Language Models
This addresses robustness issues in large language models, but appears to be a survey and proposal rather than presenting new experimental results.
The paper investigates language models' weak generalization outside training domains as a root cause of their preference for surface-level patterns over semantic complexity, finding that incorporating generalization measures into training objectives improves distributional robustness.
Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for simple, surface-level textual relations over full semantic complexity of the problem. This proposal investigates a common denominator of this problem in their weak ability to generalise outside of the training domain. We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models. Based on these findings, we present future research directions towards enhancing the robustness of LLMs.