arXiv:2510.21961v1 Announce Type: cross 
Abstract: Masked diffusion models (MDMs) offer a compelling alternative to autoregressive models (ARMs) for discrete text generation because they enable parallel token sampling, rather than sequential, left-to-right generation. This means potentially much faster inference. However, effective parallel sampling faces two competing requirements: (i) simultaneously updated tokens must be conditionally independent, and (ii) updates should prioritise high-confidence predictions. These goals conflict because high-confidence predictions often cluster and depend on each other, opportunities for parallel updates.
  We present PUNT, a model-agnostic sampler that reconciles this trade-off. Our method identifies token dependencies and removes lower-confidence tokens from conflicting groups. This produces sets of indices for unmasking that satisfy both independence and confidence criteria. Our approach ensures improved parallel unmasking through approximate conditional independence testing.
  Our experiments show that PUNT delivers a superior trade-off between accuracy and compute when compared to other strong training-free baselines, especially for generation of longer sequences. On the IFEval benchmark, it achieves up to 16\% higher accuracy over baseline methods, including sequential generation (one-by-one). These gains hold across different values of hyperparameters, mitigating the need for brittle hyperparameter tuning. Moreover, we observe that PUNT induces an emergent hierarchical generation strategy, where the model first establishes high-level paragraph structure before local refinement, suggesting a planning-like generation process that contributes to strong alignment performance.

تسلط دراسة حديثة الضوء على مزايا نماذج الانتشار المقنعة (MDMs) مقارنة بالنماذج التلقائية التقليدية (ARMs) في توليد النصوص. تتيح MDMs أخذ عينات متوازية من الرموز، مما قد يحدث ثورة في كيفية توليد النصوص. هذا مهم لأنه لا يسرع العملية فحسب، بل يحافظ أيضًا على جودة المحتوى الناتج. تؤكد الأبحاث على أهمية ضمان استقلالية الرموز بشكل شرطي مع إعطاء الأولوية للتحديثات عالية الثقة، مما يمهد الطريق لتقنيات توليد نصوص أكثر كفاءة.

Un estudio reciente destaca las ventajas de los modelos de difusión enmascarados (MDMs) sobre los modelos autorregresivos tradicionales (ARMs) en la generación de texto. Los MDMs permiten un muestreo paralelo de tokens, lo que podría revolucionar nuestra forma de generar texto. Esto es significativo porque no solo acelera el proceso, sino que también mantiene la calidad del contenido generado. La investigación enfatiza la importancia de asegurar que los tokens permanezcan condicionalmente independientes mientras se priorizan las actualizaciones de alta confianza, allanando el camino para técnicas de generación de texto más eficientes.

Une étude récente met en avant les avantages des modèles de diffusion masqués (MDMs) par rapport aux modèles autorégressifs traditionnels (ARMs) dans la génération de texte. Les MDMs permettent un échantillonnage parallèle des tokens, ce qui pourrait révolutionner notre façon de générer du texte. Cela est significatif car cela accélère non seulement le processus, mais maintient également la qualité du contenu généré. La recherche souligne l'importance de garantir que les tokens restent conditionnellement indépendants tout en priorisant les mises à jour de haute confiance, ouvrant la voie à des techniques de génération de texte plus efficaces.

A recent study highlights the advantages of masked diffusion models (MDMs) over traditional autoregressive models (ARMs) in text generation. MDMs allow for faster, parallel token sampling, which could revolutionize how we generate text. This is significant because it not only speeds up the process but also maintains the quality of the generated content. The research emphasizes the importance of ensuring that tokens remain conditionally independent while prioritizing high-confidence updates, paving the way for more efficient and effective text generation techniques.

Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing

arXiv:2502.03930v4 Announce Type: replace-cross 
Abstract: Several recent studies have attempted to autoregressively generate continuous speech representations without discrete speech tokens by combining diffusion and autoregressive models, yet they often face challenges with excessive computational loads or suboptimal outcomes. In this work, we propose Diffusion Transformer Autoregressive Modeling (DiTAR), a patch-based autoregressive framework combining a language model with a diffusion transformer. This approach significantly enhances the efficacy of autoregressive models for continuous tokens and reduces computational demands. DiTAR utilizes a divide-and-conquer strategy for patch generation, where the language model processes aggregated patch embeddings and the diffusion transformer subsequently generates the next patch based on the output of the language model. For inference, we propose defining temperature as the time point of introducing noise during the reverse diffusion ODE to balance diversity and determinism. We also show in the extensive scaling analysis that DiTAR has superb scalability. In zero-shot speech generation, DiTAR achieves state-of-the-art performance in robustness, speaker similarity, and naturalness.

يمثل تقديم DiTAR، أو نمذجة التحويل التلقائي بواسطة نموذج الانتشار، تقدمًا كبيرًا في مجال توليد الكلام من خلال دمج نموذج لغة مع نموذج انتشار. يتناول هذا الإطار المبتكر التحديات الحسابية التي واجهتها النماذج التلقائية السابقة، مما يعزز فعاليتها في توليد رموز الكلام المستمرة.

La introducción de DiTAR, o Modelado Autoregresivo de Transformador de Difusión, representa un avance significativo en el campo de la generación de voz al integrar un modelo de lenguaje con un transformador de difusión. Este marco innovador aborda los desafíos computacionales que enfrentaban los modelos autoregresivos anteriores, mejorando su eficiencia para la generación de tokens de voz continua.

L'introduction de DiTAR, ou Modélisation Autoregressive par Transformateur de Diffusion, représente une avancée significative dans le domaine de la génération de la parole en intégrant un modèle de langage avec un transformateur de diffusion. Ce cadre innovant répond aux défis computationnels rencontrés par les modèles autoregressifs précédents, améliorant leur efficacité pour la génération de tokens de parole continue.

The introduction of DiTAR, or Diffusion Transformer Autoregressive Modeling, represents a significant advancement in the field of speech generation by integrating a language model with a diffusion transformer. This innovative framework addresses the computational challenges faced by previous autoregressive models, enhancing their efficiency for continuous speech token generation.

DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation

arXiv:2508.12970v2 Announce Type: replace-cross 
Abstract: The autoregressive time series model is a popular second-order stationary process, modeling a wide range of real phenomena. However, in applications, autoregressive signals are often corrupted by additive noise. Further, the autoregressive process and the corruptive noise may be highly impulsive, stemming from an infinite-variance distribution. The model estimation techniques that account for additional noise tend to show reduced efficacy when there is very strong noise present in the data, especially when the noise is heavy-tailed. In this paper, we propose a novel self-supervised learning method to denoise the additive noise-corrupted autoregressive model. Our approach is motivated by recent work in computer vision and does not require full knowledge of the noise distribution. We use the proposed method to recover exemplary finite- and infinite-variance autoregressive signals, namely, Gaussian and alpha-stable distributed signals, respectively, from their noise-corrupted versions. The simulation study conducted on both synthetic and semi-synthetic data demonstrates strong denoising performance of our method compared to several baseline methods, particularly when the corruption is significant and impulsive in nature. Finally, we apply the presented methodology to forecast the pure autoregressive signal from the noise-corrupted data.

تم اقتراح طريقة جديدة للتعلم الذاتي المراقب لإزالة الضوضاء من النماذج الذاتية الانحدار المتأثرة بالضوضاء المضافة، حيث تتناول كل من حالات التباين المحدود وغير المحدود. يستند هذا النهج إلى رؤى من رؤية الكمبيوتر ولا يتطلب معرفة كاملة بتوزيع الضوضاء، مما يعزز استعادة الإشارات مثل التوزيعات الغاوسية والمستقرة ألفا.

Se ha propuesto un nuevo método de aprendizaje auto-supervisado para eliminar el ruido de los modelos autorregresivos afectados por ruido aditivo, abordando tanto los casos de varianza finita como infinita. Este enfoque se basa en conocimientos de la visión por computadora y no requiere un conocimiento completo de la distribución del ruido, mejorando la recuperación de señales como las distribuciones gaussiana y alfa-estable.

Une nouvelle méthode d'apprentissage auto-supervisé a été proposée pour débruiter les modèles autorégressifs affectés par du bruit additif, abordant à la fois les cas de variance finie et infinie. Cette approche s'inspire de la vision par ordinateur et ne nécessite pas une connaissance complète de la distribution du bruit, améliorant ainsi la récupération de signaux tels que les distributions gaussiennes et alpha-stables.

A novel self-supervised learning method has been proposed for denoising autoregressive models that are affected by additive noise, addressing both finite and infinite variance cases. This approach leverages insights from computer vision and does not require complete knowledge of the noise distribution, enhancing the recovery of signals such as Gaussian and alpha-stable distributions.

Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing

Was this article worth reading? Share it

LucidQuery AI

Magicley AI

Airparser