SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines
This work addresses the challenge of integrating different neural network architectures for natural language processing, offering a novel hybrid approach that is particularly useful in data-scarce scenarios.
The paper tackled the problem of bridging CNNs and RNNs for natural language encoding by introducing SoPa, a model that combines neural representation learning with weighted finite-state automata, and showed it is comparable or better than BiLSTM and CNN baselines on three text classification tasks, especially in small data settings.
Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances. In this paper we present SoPa, a new model that aims to bridge these two approaches. SoPa combines neural representation learning with weighted finite-state automata (WFSAs) to learn a soft version of traditional surface patterns. We show that SoPa is an extension of a one-layer CNN, and that such CNNs are equivalent to a restricted version of SoPa, and accordingly, to a restricted form of WFSA. Empirically, on three text classification tasks, SoPa is comparable or better than both a BiLSTM (RNN) baseline and a CNN baseline, and is particularly useful in small data settings.