Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
This addresses the need for efficient 3D token representation in robotics and vision tasks, offering an incremental improvement over existing position encoding methods.
The paper tackles the problem of position encoding for multi-dimensional tokens in transformers by introducing STRING, a separable translationally invariant position encoding method that extends Rotary Position Encodings. The result shows substantial gains in applications like open-vocabulary object detection and robotics controllers when integrated into Vision Transformers with RGB(-D) inputs.
We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides exact translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key. We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers. We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods.