arXiv:2510.25544v2 Announce Type: replace 
Abstract: Recently proposed generative models for discrete data, such as Masked Diffusion Models (MDMs), exploit conditional independence approximations to reduce the computational cost of popular Auto-Regressive Models (ARMs), at the price of some bias in the sampling distribution. We study the resulting computation-vs-accuracy trade-off, providing general error bounds (in relative entropy) that depend only on the average number of tokens generated per iteration and are independent of the data dimensionality (i.e. sequence length), thus supporting the empirical success of MDMs. We then investigate the gain obtained by using non-constant schedule sizes (i.e. varying the number of unmasked tokens during the generation process) and identify the optimal schedule as a function of a so-called information profile of the data distribution, thus allowing for a principled optimization of schedule sizes. We define methods directly as sampling algorithms and do not use classical derivations as time-reversed diffusion processes, leading us to simple and transparent proofs.

تركزت الأبحاث الأخيرة على نماذج الانتشار المقنعة (MDMs)، التي تستخدم تقريبات الاستقلال الشرطي لتعزيز الكفاءة الحسابية مقارنة بالنماذج الذاتية الانحدار (ARMs). تقدم هذه الدراسة حدود خطأ عامة مستقلة عن أبعاد البيانات، مما يبرز التوازن بين الحساب والدقة في MDMs.

Investigaciones recientes se han centrado en los Modelos de Difusión Enmascarados (MDMs), que utilizan aproximaciones de independencia condicional para mejorar la eficiencia computacional en comparación con los Modelos Auto-Regresivos (ARMs). Este estudio proporciona límites de error generales que son independientes de la dimensionalidad de los datos, enfatizando la compensación entre el cálculo y la precisión en los MDMs.

Des recherches récentes se sont concentrées sur les Modèles de Diffusion Masqués (MDMs), qui utilisent des approximations d'indépendance conditionnelle pour améliorer l'efficacité computationnelle par rapport aux Modèles Auto-Régressifs (ARMs). Cette étude fournit des bornes d'erreur générales qui sont indépendantes de la dimensionnalité des données, soulignant le compromis entre le calcul et la précision dans les MDMs.

Recent research has focused on Masked Diffusion Models (MDMs), which utilize conditional independence approximations to enhance computational efficiency compared to Auto-Regressive Models (ARMs). This study provides general error bounds that are independent of data dimensionality, emphasizing the trade-off between computation and accuracy in MDMs.

Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximations

arXiv:2601.07894v1 Announce Type: new 
Abstract: Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes and datasets are available at https://github.com/NEUIR/Attention-Floating.

كشفت دراسة حديثة عن آلية الانتباه العائم في نماذج الانتشار المقنعة (MDMs)، موضحة سلوكيات الانتباه الفريدة التي تختلف عن النماذج الذاتية التكرار التقليدية (ARMs). تُظهر هذه الأبحاث أن نماذج MDMs تستخدم نقاط انتباه ديناميكية تتحرك عبر الطبقات وخطوات إزالة الضوضاء، مما يساهم في تحسين أدائها في المهام التي تتطلب التعلم في السياق.

Un estudio reciente ha revelado el mecanismo de Atención Flotante en los Modelos de Difusión Enmascarados (MDMs), destacando sus comportamientos de atención únicos que difieren de los modelos autorregresivos tradicionales (ARMs). Esta investigación muestra que los MDMs utilizan anclas de atención dinámicas que se desplazan a través de capas y pasos de desruido, contribuyendo a su rendimiento mejorado en tareas que requieren aprendizaje en contexto.

Une étude récente a révélé le mécanisme d'Attention Floating dans les Modèles de Diffusion Masqués (MDMs), mettant en lumière leurs comportements d'attention uniques qui diffèrent des modèles autoregressifs traditionnels (ARMs). Cette recherche montre que les MDMs utilisent des ancres d'attention dynamiques qui se déplacent à travers les couches et les étapes de débruitage, contribuant à leur performance améliorée dans les tâches nécessitant un apprentissage contextuel.

A recent study has unveiled the Attention Floating mechanism in Masked Diffusion Models (MDMs), highlighting their unique attention behaviors that differ from traditional autoregressive models (ARMs). This research reveals that MDMs utilize dynamic attention anchors that shift across layers and denoising steps, contributing to their enhanced performance in tasks requiring in-context learning.

Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximations

Was this article worth reading? Share it

LucidQuery AI

Magicley AI

GPTHumanizer

AiReelGenerator.com

Https

Fakeface

Ready to build your own newsroom?