Wavelet-Induced Rotary Encodings: RoPE Meets Graphs
This provides a method for incorporating graph structure into models like LLMs and ViTs, though it appears incremental as an extension of RoPE.
The authors tackled the problem of extending Rotary Position Encodings (RoPE) from sequence and grid data to graph-structured data by introducing WIRE, which recovers RoPE as a special case and demonstrates effectiveness in tasks like identifying monochromatic subgraphs and semantic segmentation of point clouds.
We introduce WIRE: Wavelet-Induced Rotary Encodings. WIRE extends Rotary Position Encodings (RoPE), a popular algorithm in LLMs and ViTs, to graph-structured data. We demonstrate that WIRE is more general than RoPE, recovering the latter in the special case of grid graphs. WIRE also enjoys a host of desirable theoretical properties, including equivariance under node ordering permutation, compatibility with linear attention, and (under select assumptions) asymptotic dependence on graph resistive distance. We test WIRE on a range of synthetic and real-world tasks, including identifying monochromatic subgraphs, semantic segmentation of point clouds, and more standard graph benchmarks. We find it to be effective in settings where the underlying graph structure is important.