arXiv:2605.07210v2 Announce Type: replace-cross 
Abstract: This paper shows how diffusion language models (DLMs) can be used as effective and efficient retrievers. Existing DLM-based retrievers (e.g., DiffEmbed) follow BERT-style encoding, representing each query or passage as a single mean-pooled vector. This ignores how DLMs are trained to generate responses through masked-position prediction under bidirectional attention, a capability that can provide stronger retrieval signals. We propose DiffRetriever, which uses the DLM's native masked-position prediction directly for retrieval. For each query or passage, DiffRetriever appends one or more masked positions, using the outputs as retrieval representations in a single forward pass. With one masked position, single-representation DiffRetriever already improves over DiffEmbed on the same backbones. DiffRetriever also naturally extends to multi-representation retrieval: DLMs process multiple masked positions jointly, enabling ColBERT-style fine-grained matching with little additional encoding latency. In autoregressive LLM retrievers, the same multi-representation strategy requires sequential decoding and therefore incurs much higher latency. DiffRetriever obtains the strongest aggregate effectiveness within our matched comparison, outperforming DiffEmbed, PromptReps, and RepLLaMA. Masked-position counts selected on training data transfer well across datasets, while per-query variation suggests headroom for adaptive allocation. Code is available at https://github.com/ielab/diffretriever.

يمثل تقديم DiffRetriever تقدمًا كبيرًا في استخدام نماذج اللغة الانتشارية (DLM) لمهام الاسترجاع، مستفيدًا من قدراتها في التنبؤ بالمواقع المخفية لتحسين كفاءة الاسترجاع. على عكس النماذج السابقة التي استخدمت متجهات متوسطة، يستخدم DiffRetriever مواقع مخفية متعددة لتوليد تمثيلات استرجاع أكثر قوة في تمريرة واحدة للأمام.

La introducción de DiffRetriever marca un avance significativo en el uso de modelos de lenguaje de difusión (DLM) para tareas de recuperación, aprovechando sus capacidades de predicción de posiciones enmascaradas para mejorar la eficiencia de la recuperación. A diferencia de los modelos anteriores que utilizaban vectores promediados, DiffRetriever emplea múltiples posiciones enmascaradas para generar representaciones de recuperación más robustas en un solo pase hacia adelante.

L'introduction de DiffRetriever représente une avancée significative dans l'utilisation des modèles de langage de diffusion (DLM) pour les tâches de récupération, en tirant parti de leurs capacités de prédiction de positions masquées pour améliorer l'efficacité de la récupération. Contrairement aux modèles précédents qui utilisaient des vecteurs moyennés, DiffRetriever emploie plusieurs positions masquées pour générer des représentations de récupération plus robustes en un seul passage avant.

The introduction of DiffRetriever marks a significant advancement in the use of diffusion language models (DLMs) for retrieval tasks, leveraging their masked-position prediction capabilities to enhance retrieval efficiency. Unlike previous models that utilized mean-pooled vectors, DiffRetriever employs multiple masked positions to generate more robust retrieval representations in a single forward pass.

DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models

arXiv:2606.13275v1 Announce Type: new 
Abstract: This paper presents Custom ZeroCLIP, a retrieval-augmented vision-language framework for zero-shot captioning of Indonesian traditional garments. The dataset contains 3,800 expert-annotated images from all 38 Indonesian provinces. Using a province-level inductive zero-shot protocol, the model is trained on 24 seen provinces, validated on 6 seen provinces, and evaluated on 8 unseen provinces. The framework combines a frozen CLIP ViT-B/32 image encoder, a CLIP text encoder, a BERT text encoder, and an LSTM caption decoder. During inference, unseen-province labels and captions are unavailable, and retrieval uses only captions from training provinces. No unseen-province image, label, or caption is used during training, validation, or retrieval-bank construction. Custom ZeroCLIP achieves a CLIPScore of 0.8536, BLEU-4 of 0.3342, and METEOR of 0.4859, outperforming existing baselines. Ablation results show that retrieval improves cultural vocabulary recovery with a 19.3\% METEOR gain, while human evaluation confirms stronger cultural accuracy and fluency. The results demonstrate the effectiveness of retrieval-augmented domain adaptation for culturally grounded caption generation in low-resource heritage settings. The dataset is publicly available at https://github.com/AnugrahAidinYotolembah/Traditional-Indonesian-Clothing-Captioning-Dataset.

تم تطوير إطار جديد يسمى Custom ZeroCLIP لتوليد تسميات بدون تدريب مسبق للملابس التقليدية الإندونيسية، باستخدام مجموعة بيانات تحتوي على 3,800 صورة تم وضع علامات عليها من قبل خبراء من جميع المحافظات الإندونيسية الـ 38. يستخدم النموذج مزيجًا من التقنيات المتقدمة، بما في ذلك مشفر صورة CLIP ViT-B/32 المجمد ومشفر نص BERT، محققًا مقاييس أداء ملحوظة مثل CLIPScore بقيمة 0.8536 وBLEU-4 بقيمة 0.3342.

Se ha desarrollado un nuevo marco llamado Custom ZeroCLIP para la creación de leyendas en cero disparos de prendas tradicionales indonesias, utilizando un conjunto de datos de 3,800 imágenes anotadas por expertos de las 38 provincias indonesias. El modelo emplea una combinación de tecnologías avanzadas, incluyendo un codificador de imagen CLIP ViT-B/32 congelado y un codificador de texto BERT, logrando métricas de rendimiento notables como un CLIPScore de 0.8536 y un BLEU-4 de 0.3342.

Un nouveau cadre appelé Custom ZeroCLIP a été développé pour la légende zéro-shot des vêtements traditionnels indonésiens, utilisant un ensemble de données de 3 800 images annotées par des experts provenant des 38 provinces indonésiennes. Le modèle emploie une combinaison de technologies avancées, y compris un encodeur d'image CLIP ViT-B/32 gelé et un encodeur de texte BERT, atteignant des métriques de performance notables telles qu'un CLIPScore de 0,8536 et un BLEU-4 de 0,3342.

A new framework called Custom ZeroCLIP has been developed for zero-shot captioning of Indonesian traditional garments, utilizing a dataset of 3,800 expert-annotated images from all 38 Indonesian provinces. The model employs a combination of advanced technologies, including a frozen CLIP ViT-B/32 image encoder and a BERT text encoder, achieving notable performance metrics such as a CLIPScore of 0.8536 and BLEU-4 of 0.3342.

Zero-Shot Captioning for Cultural Heritage: Automated Image Analysis of Traditional Indonesian Clothing

arXiv:2606.11860v1 Announce Type: new 
Abstract: In this paper, we introduce Representation Prediction via Autoencoding using Iterative Refinement (RePAIR) - a novel self-supervised representation learning architecture that synthesizes Masked Autoencoders (MAE), Joint Embedding Predictive Architectures (JEPA), and Bidirectional Encoder Representations from Transformers (BERT). We demonstrate how it can be used to encode objects in sequential data like consecutive chess positions into compact yet meaningful representations. The basic principle of the architecture is to mask large portions of a sequence of latent states, similar to BERT and MAE. Then, we apply a lightweight Predictor to the latent representations that repairs gaps in the sequence in a lower-dimensional embedding space akin to JEPA. Our experiments in the domain of chess show that the Encoder refines the board representations such that meaningful chess concepts emerge clustered in the latent space. Furthermore, reconstructions of the masked board states show that the model is able to reason about the piece movements without relying on costly reinforcement learning methods. Lastly, we find that the resulting representation space allows for quick and intuitive dissections of chess games by observing the game path trajectories in this semantically rich space.

تم تقديم بنية جديدة للتعلم الذاتي المراقب تُدعى RePAIR، والتي تدمج مفاهيم من أجهزة التشفير المقنعة (MAE) وعمارة التضمين التنبؤية المشتركة (JEPA) وBERT لتشفير البيانات التسلسلية، وخاصة في الشطرنج. تقوم هذه البنية بإخفاء أجزاء من الحالات الكامنة وتستخدم مُتنبئًا خفيف الوزن لملء الفجوات، مما يؤدي إلى تمثيلات مصقولة لمواقع الشطرنج.

Se ha introducido una nueva arquitectura de aprendizaje de representación auto-supervisada llamada RePAIR, que integra conceptos de Autoencoders enmascarados (MAE), Arquitecturas Predictivas de Embedding Conjunto (JEPA) y BERT para codificar datos secuenciales, específicamente en ajedrez. Esta arquitectura enmascara porciones de estados latentes y emplea un Predictor ligero para llenar los vacíos, resultando en representaciones refinadas de las posiciones de ajedrez.

Une nouvelle architecture d'apprentissage de représentation auto-supervisée appelée RePAIR a été introduite, intégrant des concepts de Masquage d'Autoencodeurs (MAE), d'Architectures Prédictives d'Embarquement Commun (JEPA) et de BERT pour encoder des données séquentielles, en particulier aux échecs. Cette architecture masque des portions d'états latents et utilise un Prédicteur léger pour combler les lacunes, aboutissant à des représentations affinées des positions d'échecs.

A new self-supervised representation learning architecture called RePAIR has been introduced, which integrates concepts from Masked Autoencoders (MAE), Joint Embedding Predictive Architectures (JEPA), and BERT to encode sequential data, specifically in chess. This architecture masks portions of latent states and employs a lightweight Predictor to fill in gaps, resulting in refined representations of chess positions.

DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models

Was this article worth reading? Share it

Ready to build your own newsroom?