Geotokens and Geotransformers
Eren Unlu
In transformer architectures, position encoding primarily provides a sense of
sequence for input tokens. While the original transformer paper's method has
shown satisfactory results in general language processing tasks, there have
been new proposals, such as Rotary Position Embedding (RoPE), for further
improvement. This paper presents geotokens, input components for transformers,
each linked to a specific geological location. Unlike typical language
sequences, for these tokens, the order is not as vital as the geographical
coordinates themselves. To represent the relative position in this context and
to keep a balance between the real world distance and the distance in the
embedding space, we design a position encoding approach drawing from the RoPE
structure but tailored for spherical coordinates.