The role of positional encodings in the ARC benchmark
This work addresses a specific bottleneck in AI systems for abstract reasoning tasks, offering incremental improvements for researchers and practitioners in machine learning.
The paper tackled the problem of positional encoding limitations in transformer models performing abstract reasoning on the ARC benchmark, finding that 2D positional encoding outperforms alternatives like Rotary Position Embedding in data-constrained scenarios.
The Abstraction and Reasoning Corpus challenges AI systems to perform abstract reasoning with minimal training data, a task intuitive for humans but demanding for machine learning models. Using CodeT5+ as a case study, we demonstrate how limitations in positional encoding hinder reasoning and impact performance. This work further examines the role of positional encoding across transformer architectures, highlighting its critical influence on models of varying sizes and configurations. Comparing several strategies, we find that while 2D positional encoding and Rotary Position Embedding offer competitive performance, 2D encoding excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks