arXiv:2606.10650v1 Announce Type: new 
Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.

تم اقتراح إطار جديد يسمى Dynamic Linear Attention (DLA) لتحسين قابلية التوسع لنماذج اللغة الكبيرة (LLMs) في معالجة السياقات الطويلة من خلال تقديم دمج ديناميكي للحالة الواعية بالمعلومات. تهدف هذه الطريقة إلى تحسين قدرة التمثيل من خلال تحديد حدود الحالة بشكل تكيفي بناءً على أهمية الرموز، مما يعالج قيود طرق الانتباه متعددة الحالات الحالية.

Se ha propuesto un nuevo marco llamado Dynamic Linear Attention (DLA) para mejorar la escalabilidad de los Modelos de Lenguaje Grande (LLMs) en el procesamiento de contextos largos, introduciendo la Fusión Dinámica de Estados Consciente de la Información. Este método busca mejorar la capacidad de representación al determinar de manera adaptativa los límites de estado según la importancia de los tokens, abordando así las limitaciones de los métodos de atención lineal multiestado existentes.

Un nouveau cadre appelé Dynamic Linear Attention (DLA) a été proposé pour améliorer l'évolutivité des grands modèles de langage (LLMs) dans le traitement de longs contextes en introduisant le regroupement d'état dynamique conscient de l'information. Cette méthode vise à améliorer la capacité de représentation en déterminant de manière adaptative les limites d'état en fonction de l'importance des jetons, abordant ainsi les limitations des méthodes d'attention linéaire multi-état existantes.

A new framework called Dynamic Linear Attention (DLA) has been proposed to enhance the scalability of Large Language Models (LLMs) in processing long contexts by introducing Information-Aware Dynamic State Merging. This method aims to improve representation capacity by adaptively determining state boundaries based on token importance, addressing limitations of existing multi-state linear attention methods.