Subword models struggle with word learning, but surprisal hides it
This research addresses the problem of modeling language acquisition in computational linguistics, suggesting subword LMs may be inadequate and character LMs as an alternative, though it is incremental in comparing existing model types.
The study investigated word learning in subword and character language models using a psycholinguistic lexical decision task, finding that character LMs easily discern words from non-words while subword LMs struggle without additional context, and that word and syntactic learning are separable in character LMs but simultaneous in subword LMs.
We study word learning in subword and character language models with the psycholinguistic lexical decision task. While subword LMs struggle to discern words and non-words with high accuracy, character LMs solve this task easily and consistently. Only when supplied with further contexts do subword LMs perform similarly to character models. Additionally, when looking at word-level and syntactic learning trajectories, we find that both processes are separable in character LMs. Word learning happens before syntactic learning, whereas both occur simultaneously in subword LMs. This raises questions about the adequacy of subword LMs for modeling language acquisition and positions character LMs as a viable alternative to study processes below the syntactic level.