Learning to Learn to be Right for the Right Reasons
This addresses the issue of robust generalization in commonsense reasoning for AI systems, though it is incremental as it builds on existing methods to handle superficial cues.
The paper tackles the problem of models overfitting to superficial cues in commonsense reasoning, which leads to poor generalization on hard test sets without such cues, and proposes a meta-learning method that improves performance on both easy and hard test sets, achieving up to 16.5 percentage points improvement over baselines.
Improving model generalization on held-out data is one of the core objectives in commonsense reasoning. Recent work has shown that models trained on the dataset with superficial cues tend to perform well on the easy test set with superficial cues but perform poorly on the hard test set without superficial cues. Previous approaches have resorted to manual methods of encouraging models not to overfit to superficial cues. While some of the methods have improved performance on hard instances, they also lead to degraded performance on easy instances. Here, we propose to explicitly learn a model that does well on both the easy test set with superficial cues and hard test set without superficial cues. Using a meta-learning objective, we learn such a model that improves performance on both the easy test set and the hard test set. By evaluating our models on Choice of Plausible Alternatives (COPA) and Commonsense Explanation, we show that our proposed method leads to improved performance on both the easy test set and the hard test set upon which we observe up to 16.5 percentage points improvement over the baseline.