Embedding Lexical Features via Low-Rank Tensors
This addresses overfitting and efficiency issues in NLP models that use complex lexical features, though it appears incremental as it builds on existing tensor methods.
The paper tackles the problem of overfitting from large numbers of engineered lexical features in NLP by representing them in a tensor and applying low-rank approximations to reduce parameters and improve speed, achieving state-of-the-art results on tasks like relation extraction, PP-attachment, and preposition disambiguation.
Modern NLP models rely heavily on engineered features, which often combine word and contextual information into complex lexical features. Such combination results in large numbers of features, which can lead to over-fitting. We present a new model that represents complex lexical features---comprised of parts for words, contextual information and labels---in a tensor that captures conjunction information among these parts. We apply low-rank tensor approximations to the corresponding parameter tensors to reduce the parameter space and improve prediction speed. Furthermore, we investigate two methods for handling features that include $n$-grams of mixed lengths. Our model achieves state-of-the-art results on tasks in relation extraction, PP-attachment, and preposition disambiguation.