Generalising Recursive Neural Models by Tensor Decomposition
This work addresses a bottleneck in recursive neural models for structured data, offering a method to enhance expressivity and generalization, though it is incremental in nature.
The authors tackled the suboptimality of simple aggregation functions in neural models for structured data by introducing a tensor-based formulation with Tucker decomposition to control parameter growth, achieving improved performance in tree classification tasks.
Most machine learning models for structured data encode the structural knowledge of a node by leveraging simple aggregation functions (in neural models, typically a weighted sum) of the information in the node's neighbourhood. Nevertheless, the choice of simple context aggregation functions, such as the sum, can be widely sub-optimal. In this work we introduce a general approach to model aggregation of structural context leveraging a tensor-based formulation. We show how the exponential growth in the size of the parameter space can be controlled through an approximation based on the Tucker tensor decomposition. This approximation allows limiting the parameters space size, decoupling it from its strict relation with the size of the hidden encoding space. By this means, we can effectively regulate the trade-off between expressivity of the encoding, controlled by the hidden size, computational complexity and model generalisation, influenced by parameterisation. Finally, we introduce a new Tensorial Tree-LSTM derived as an instance of our framework and we use it to experimentally assess our working hypotheses on tree classification scenarios.