APE

“Though simple and straightforward, APE-based Transformers usually generalize poorly to longer sequences”

“Although both position embeddings are effective for the transformer on fixed-resolution settings, they struggle with resolution changes, requiring flexibility and extrapolation in position embeddings.”

“APE has well-documented limitations: it struggles to generalize to resolutions unseen during training and provides no explicit mechanism for encoding relative spatial relationships between image patches.”

“A key limitation of APE methods is their poor generalization to sequence lengths beyond those seen during training, making them unsuitable for length extrapolation.”

“neither the learnable nor the fixed sinusoidal embedding can generalize well to longer sequences”

“the fixed nature of positional encoding limited the model's ability to generalize to longer input sequences”

“Absolute Positional Encodings(APE)~vaswani2017attention, which utilize sine and cosine functions, are inadequate for length extrapolation.”

“Existing absolute position encoding (APE) vaswani2017attention, devlin2018bert incorporates either fixed or learnable position encodings into input representations through vector addition. However, APE faces challenges when dealing with long-contexts.”