Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models
This work addresses the problem of overfitting and poor generalization in commonsense reasoning for AI researchers, though it is incremental as it builds on existing lightweight update strategies.
The paper investigates how models learn from commonsense reasoning datasets by measuring the impact of three adaptation methods, finding that fine-tuning achieves the best accuracy but suffers from overfitting and limited generalization to novel answers, while methods like prefix-tuning offer comparable accuracy with better generalization and robustness.
Commonsense reasoning benchmarks have been largely solved by fine-tuning language models. The downside is that fine-tuning may cause models to overfit to task-specific data and thereby forget their knowledge gained during pre-training. Recent works only propose lightweight model updates as models may already possess useful knowledge from past experience, but a challenge remains in understanding what parts and to what extent models should be refined for a given task. In this paper, we investigate what models learn from commonsense reasoning datasets. We measure the impact of three different adaptation methods on the generalization and accuracy of models. Our experiments with two models show that fine-tuning performs best, by learning both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers. We observe that alternative adaptation methods like prefix-tuning have comparable accuracy, but generalize better to unseen answers and are more robust to adversarial splits.