Do We Need Neural Models to Explain Human Judgments of Acceptability?
This work addresses the need for efficient and interpretable models in computational linguistics to evaluate language processing, though it is incremental by showing that simple features can match neural approaches.
The study tackled the problem of predicting human judgments of sentence acceptability in English essays by non-native speakers, finding that a 4-gram model with misspelling counts achieved a Pearson's r of 0.528, surpassing previous unsupervised methods and matching neural models.
Native speakers can judge whether a sentence is an acceptable instance of their language. Acceptability provides a means of evaluating whether computational language models are processing language in a human-like manner. We test the ability of computational language models, simple language features, and word embeddings to predict native English speakers judgments of acceptability on English-language essays written by non-native speakers. We find that much of the sentence acceptability variance can be captured by a combination of features including misspellings, word order, and word similarity (Pearson's r = 0.494). While predictive neural models fit acceptability judgments well (r = 0.527), we find that a 4-gram model with statistical smoothing is just as good (r = 0.528). Thanks to incorporating a count of misspellings, our 4-gram model surpasses both the previous unsupervised state-of-the art (Lau et al., 2015; r = 0.472), and the average non-expert native speaker (r = 0.46). Our results demonstrate that acceptability is well captured by n-gram statistics and simple language features.