arXiv:2605.28769v1 Announce Type: new 
Abstract: Softmax attention is the cornerstone of modern large language models, but its memory scales linearly and compute quadratically with sequence length. Linear recurrent models, such as linear attention and state space models, have become widely studied as alternatives to attention due to their linear compute and constant memory. While these sub-quadratic token mixing methods, or mixers, achieve promising efficiency gains and competitive results on a wide range of benchmarks, current linear recurrent models still lag behind on tasks that require long-context retrieval or in-context learning. A growing body of work studies hybrid architectures that attempt to mitigate these trade-offs by statically interleaving or merging attention and recurrent blocks. In this work, we explore a new axis of developing hybrid models: across the token sequence. We propose Oryx, a hybrid model that can, throughout a sequence, flexibly switch between different mixers, for example quadratic attention for rich context utilization and linear recurrences for efficient generation. Oryx ties at least 90% of its parameters across mixers, enabling attention and recurrent modes to operate over shared internal representations. We validate our design with Mamba-2 and Gated DeltaNet variants, up to 1.4B models. Under fixed token budgets and a mixed-training strategy, Oryx achieves comparable or better performance than its single-mixer baselines. At the 1.4B scale, all instances of Oryx outperform their respective baselines by at least 0.7 percentage points on averaged language modeling tasks. On retrieval tasks, Oryx achieves performance comparable to the Transformer baseline even when processing only a tiny fraction (<10%) of the tokens in attention mode. These results suggest that attention and linear recurrent models can share internal representations, and motivate sequence-axis hybridization as a promising direction.

تمثل المقدمة الحديثة لـ Oryx، وهو نموذج هجين لنمذجة التسلسلات، تقدمًا كبيرًا في مجال الذكاء الاصطناعي. يسمح هذا النموذج بالتبديل المرن بين طرق الخلط المختلفة على مدار تسلسل الرموز، مما يعالج قيود آليات الانتباه التقليدية مثل softmax، التي تتطلب موارد حسابية وذاكرة كبيرة.

La reciente introducción de Oryx, un modelo híbrido para la modelación de secuencias, representa un avance significativo en el campo de la inteligencia artificial. Este modelo permite un cambio flexible entre diferentes métodos de mezcla a lo largo de una secuencia de tokens, abordando las limitaciones de los mecanismos de atención softmax tradicionales, que son costosos en términos de computación y memoria.

L'introduction récente d'Oryx, un modèle hybride pour la modélisation de séquences, représente une avancée significative dans le domaine de l'intelligence artificielle. Ce modèle permet un passage flexible entre différentes méthodes de mélange tout au long d'une séquence de tokens, répondant aux limitations des mécanismes d'attention softmax traditionnels, qui sont coûteux en calcul et en mémoire.

The recent introduction of Oryx, a hybrid model for sequence modeling, represents a significant advancement in the field of artificial intelligence. This model allows for flexible switching between different mixing methods throughout a token sequence, addressing the limitations of traditional softmax attention mechanisms, which are computationally expensive and memory-intensive.