arXiv:2602.18196v5 Announce Type: replace 
Abstract: Structured dilated attention has an appealing inference-time efficiency knob: it reduces the FLOPs of attention and the KV cache size by a factor of the dilation size D, while preserving long-range connectivity. While prior work studies it by training each configuration from scratch, directly sparsifying a pretrained attention model into a dilated pattern leads to severe accuracy degradation, preventing flexible reuse across inference scenarios. We introduce RAT+, a dense-pretraining architecture that augments attention with full-sequence recurrence and active recurrence learning. A single RAT+ model is pretrained densely once and can then be flexibly switched at inference time to dilated attention (optionally with local windows) or hybrid layer/head compositions, requiring only a short 1B-token resolution adaptation rather than retraining separate sparse models. At 1.5B parameters trained on 100B tokens, RAT+ closely matches dense accuracy at D = 16, and drops by about 2-3 points at D = 64 on commonsense reasoning and LongBench tasks. We further scale to 2.6B and 7.6B parameters and observe even more promising performance (e.g., a 1-point average accuracy loss with a 64x reduction in attention FLOPs and KV cache size). Code is available at https://github.com/wimh966/rat-plus.

يمثل تقديم RAT+ تقدمًا كبيرًا في آليات الانتباه، حيث يسمح بانتباه موسع منظم يعزز كفاءة وقت الاستدلال مع الحفاظ على الاتصال على المدى الطويل. يمكن تدريب هذا النموذج بشكل كثيف وتكييفه لمجموعة متنوعة من سيناريوهات الاستدلال دون الحاجة إلى إعادة تدريب نماذج منفصلة.

La introducción de RAT+ representa un avance significativo en los mecanismos de atención, permitiendo una atención dilatada estructurada que mejora la eficiencia en el tiempo de inferencia mientras mantiene la conectividad a largo alcance. Este modelo puede ser preentrenado de manera densa y adaptado a diversos escenarios de inferencia sin necesidad de reentrenar modelos separados.

L'introduction de RAT+ représente une avancée significative dans les mécanismes d'attention, permettant une attention dilatée structurée qui améliore l'efficacité en temps d'inférence tout en maintenant la connectivité à longue portée. Ce modèle peut être préentraîné de manière dense et adapté à divers scénarios d'inférence sans nécessiter de réentraînement de modèles séparés.

The introduction of RAT+ represents a significant advancement in attention mechanisms, allowing for structured dilated attention that enhances inference-time efficiency while maintaining long-range connectivity. This model can be pretrained densely and adapted for various inference scenarios without the need for retraining separate models.