DoPE: Denoising Rotary Position Embedding

The introduction of Denoising Positional Encoding (DoPE) marks a significant advancement in addressing the limitations of Rotary Position Embedding (RoPE) within Transformer models. The inherent issues with RoPE, particularly in length extrapolation, have been a barrier to effective AI performance. DoPE reinterprets the attention map as a noisy feature map and employs a training-free approach based on truncated matrix entropy to identify outlier frequency bands. This innovative method not only mitigates the attention sink phenomenon but also restores balanced attention patterns, thereby enhancing retrieval accuracy and reasoning stability across extended contexts, demonstrated through experiments on needle-in-a-haystack and many-shot in-context learning tasks. The results indicate that DoPE can effectively handle contexts of up to 64K tokens, providing a simple yet powerful solution for improving length generalization in AI models.