Geotokens and Geotransformers
This work addresses a domain-specific problem in geology or geospatial analysis by adapting transformers for non-sequential data, but it is incremental as it builds on existing RoPE methods.
The paper tackles the problem of representing geographical coordinates in transformer architectures by introducing geotokens linked to geological locations, resulting in a position encoding method based on Rotary Position Embedding tailored for spherical coordinates to balance real-world and embedding space distances.
In transformer architectures, position encoding primarily provides a sense of sequence for input tokens. While the original transformer paper's method has shown satisfactory results in general language processing tasks, there have been new proposals, such as Rotary Position Embedding (RoPE), for further improvement. This paper presents geotokens, input components for transformers, each linked to a specific geological location. Unlike typical language sequences, for these tokens, the order is not as vital as the geographical coordinates themselves. To represent the relative position in this context and to keep a balance between the real world distance and the distance in the embedding space, we design a position encoding approach drawing from the RoPE structure but tailored for spherical coordinates.