A Simple Recurrent Unit with Reduced Tensor Product Representations
This work addresses the need for more interpretable and structurally aware recurrent units in natural language processing, though it appears incremental as it builds on existing TPR concepts.
The authors tackled the problem of recurrent neural networks' limited ability to learn structured representations by proposing the TPRU, a simple recurrent unit using reduced Tensor Product Representations, which showed effectiveness on multiple datasets and demonstrated interpretability in a linguistically grounded study.
idely used recurrent units, including Long-short Term Memory (LSTM) and the Gated Recurrent Unit (GRU), perform well on natural language tasks, but their ability to learn structured representations is still questionable. Exploiting reduced Tensor Product Representations (TPRs) --- distributed representations of symbolic structure in which vector-embedded symbols are bound to vector-embedded structural positions --- we propose the TPRU, a simple recurrent unit that, at each time step, explicitly executes structural-role binding and unbinding operations to incorporate structural information into learning. A gradient analysis of our proposed TPRU is conducted to support our model design, and its performance on multiple datasets shows the effectiveness of our design choices. Furthermore, observations on a linguistically grounded study demonstrate the interpretability of our TPRU.