THG: Transformer with Hyperbolic Geometry
This work addresses a domain-specific problem for researchers and practitioners in natural language processing by offering an incremental improvement over existing Transformer architectures through the integration of hyperbolic geometry.
The authors tackled the limitation of existing Transformer variants operating solely in Euclidean space by proposing a Transformer with Hyperbolic Geometry (THG) model that leverages both Euclidean and hyperbolic spaces, improving linear transformations in self-attention to enhance performance across sequence labeling, machine reading comprehension, and classification tasks while alleviating overfitting.
Transformer model architectures have become an indispensable staple in deep learning lately for their effectiveness across a range of tasks. Recently, a surge of "X-former" models have been proposed which improve upon the original Transformer architecture. However, most of these variants make changes only around the quadratic time and memory complexity of self-attention, i.e. the dot product between the query and the key. What's more, they are calculate solely in Euclidean space. In this work, we propose a novel Transformer with Hyperbolic Geometry (THG) model, which take the advantage of both Euclidean space and Hyperbolic space. THG makes improvements in linear transformations of self-attention, which are applied on the input sequence to get the query and the key, with the proposed hyperbolic linear. Extensive experiments on sequence labeling task, machine reading comprehension task and classification task demonstrate the effectiveness and generalizability of our model. It also demonstrates THG could alleviate overfitting.