André Jonasson

André Jonasson

Transformer-based Large Language Models (LLMs) rely on positional encodings to provide sequence position information to their attention mechanism. Rotary Positional Encodings (RoPE), which encode relative position by rotating queries and keys, have become widely used in modern LLMs. We study the features and patterns that emerge in queries and keys when using rotary embeddings and introduce the concept of rotary offset features. Our analysis reveals that these features, which frequently exhibit large activations and are often interpreted as outliers, arise consistently across layers, attention heads, and model architectures. We derive bounds predicting which rotary frequencies give rise to rotary offset features and the minimum angle between the query-key pairs for these features. We verify our predictions empirically across models of different sizes and architectures.

Daniel Armstrong, André Jonasson

In this paper, we present a new algorithm that extends RRT* and RT-RRT* for online path planning in complex, dynamic environments. Sampling-based approaches often perform poorly in environments with narrow passages, a feature common to many indoor applications of mobile robots as well as computer games. Our method extends RRT-based sampling methods to enable the use of an assisting distance metric to improve performance in environments with obstacles. This assisting metric, which can be any metric that has better properties than the Euclidean metric when line of sight is blocked, is used in combination with the standard Euclidean metric in such a way that the algorithm can reap benefits from the assisting metric while maintaining the desirable properties of previous RRT variants - namely probabilistic completeness in tree coverage and asymptotic optimality in path length. We also introduce a new method of targeted rewiring, aimed at shortening search times and path lengths in tasks where the goal shifts repeatedly. We demonstrate that our method offers considerable improvements over existing multi-query planners such as RT-RRT* when using diffusion distance as an assisting metric; finding near-optimal paths with a decrease in search time of several orders of magnitude. Experimental results show planning times reduced by 99.5% and path lengths by 9.8% over existing real-time RRT planners in a variety of environments.