arXiv:2602.02259v2 Announce Type: replace 
Abstract: Latent action models (LAMs) offer a promising path to pre-training embodied agents on large amounts of action-free video. They infer latent actions between consecutive observations that can later be decoded to ground-truth actions using a small number of labels. However, recent work has shown that this recipe fails in the presence of action-correlated visual distractors common in real-world video, such as dynamic backgrounds, camera shake, or other moving objects. In these scenarios, the standard reconstruction objective drives latent actions to encode exogenous motion instead of agent-controlled dynamics, resulting in policies that underperform when fine-tuned. We observe, however, that endogenous and exogenous factors are typically spatially separated in pixel space: control-relevant change is concentrated on the agent, while distractor motion occurs elsewhere. We exploit this observation by restricting the reconstruction objective to agent pixels, forcing latent actions to explain agent-controlled dynamics rather than exogenous ones. We call this method MaskLAM; it obtains the agent mask zero-shot from off-the-shelf segmentation foundation models (e.g., SAM) and requires no architectural changes, auxiliary losses, or action labels during pre-training. Across two continuous-control benchmarks (Distracting Control Suite, Distracting Meta-World), MaskLAM reduces normalized linear-probe MSE by up to $3.51\times$ and improves normalized return by up to $4.97\times$ over LAPO, while narrowing the gap to LAOM-Labels, which relies on ground-truth action supervision.

أظهرت التطورات الأخيرة في نماذج العمل الكامنة (LAM) إمكاناتها في تدريب الوكلاء المجسدين باستخدام مقاطع الفيديو الخالية من العمل. ومع ذلك، تظهر التحديات عندما تواجه هذه النماذج المشتتات البصرية المرتبطة بالعمل، مثل الخلفيات الديناميكية والأجسام المتحركة، مما قد يؤدي إلى أداء دون المستوى الأمثل عند ضبطها.

Los avances recientes en modelos de acción latente (LAM) han destacado su potencial para el preentrenamiento de agentes incorporados utilizando video sin acción. Sin embargo, surgen desafíos cuando estos modelos se encuentran con distractores visuales correlacionados con la acción, como fondos dinámicos y objetos en movimiento, lo que puede llevar a un rendimiento subóptimo en el ajuste fino.

Des avancées récentes dans les modèles d'action latents (LAM) ont mis en évidence leur potentiel pour le pré-entraînement d'agents incarnés à l'aide de vidéos sans action. Cependant, des défis se posent lorsque ces modèles rencontrent des distracteurs visuels corrélés à l'action, tels que des arrière-plans dynamiques et des objets en mouvement, ce qui peut entraîner des performances sous-optimales lors de l'affinage.

Recent advancements in latent action models (LAMs) have highlighted their potential for pre-training embodied agents using action-free video. However, challenges arise when these models encounter action-correlated visual distractors, such as dynamic backgrounds and moving objects, which can lead to suboptimal performance in fine-tuning.

Segment to Focus: Guiding Latent Action Models in the Presence of Distractors

arXiv:2606.07100v1 Announce Type: new 
Abstract: Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.

تهدف مقدمة أداة محاذاة تمثيل العمل الكامن (LARA) إلى تحسين تدريب نماذج العمل-اللغة-الرؤية (VLA) من خلال تحسين نماذج العمل الكامنة (LAM) وVLA بشكل مشترك عبر محاذاة التمثيل، مما يعالج القيود في جودة البيانات وتوافرها.

La introducción del Alineamiento de Representación de Acción Latente (LARA) tiene como objetivo mejorar el entrenamiento de los modelos de Acción-Lenguaje-Visual (VLA) al optimizar conjuntamente los Modelos de Acción Latente (LAM) y VLA a través del alineamiento de representaciones, abordando las limitaciones en la calidad y disponibilidad de datos.

L'introduction de l'Alignement de Représentation d'Action Latente (LARA) vise à améliorer l'entraînement des modèles d'Action-Vision-Langage (VLA) en optimisant conjointement les Modèles d'Action Latente (LAM) et VLA via un alignement de représentation, répondant aux limitations de la qualité et de la disponibilité des données.

The introduction of Latent Action Representation Alignment (LARA) aims to enhance the training of Vision-Language-Action (VLA) models by jointly optimizing Latent Action Models (LAM) and VLA through representation alignment, addressing limitations in data quality and availability.

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

arXiv:2602.19213v2 Announce Type: replace 
Abstract: Medical image segmentation is vital for clinical diagnosis and quantitative analysis, yet remains challenging due to the heterogeneity of imaging modalities and the high cost of pixel-level annotations. Although general interactive segmentation models like SAM have achieved remarkable progress, their transfer to medical imaging still faces two key bottlenecks: (i) the lack of adaptive mechanisms for modality- and anatomy-specific tasks, which limits generalization in out-of-distribution medical scenarios; and (ii) current medical adaptation methods fine-tune on large, heterogeneous datasets without selection, leading to noisy supervision, higher cost, and negative transfer. To address these issues, we propose SegMoTE, an efficient and adaptive framework for medical image segmentation. SegMoTE preserves SAM's original prompt interface, efficient inference, and zero-shot generalization while introducing only a small number of learnable parameters to dynamically adapt across modalities and tasks. In addition, we design a progressive prompt tokenization mechanism that enables fully automatic segmentation, significantly reducing annotation dependence. Trained on MedSeg-HQ, a curated dataset less than 1% of existing large-scale datasets, SegMoTE achieves SOTA performance across diverse imaging modalities and anatomical tasks. It represents the first efficient, robust, and scalable adaptation of general segmentation models to the medical domain under extremely low annotation cost, advancing the practical deployment of foundation vision models in clinical applications.

تمثل مقدمة SegMoTE، وهو إطار عمل لمزيج من الخبراء على مستوى الرموز لقطاع الصور الطبية، استجابة لتحديات كبيرة في هذا المجال، خاصةً قيود النماذج الحالية مثل SAM في التكيف مع أنماط التصوير الطبي المتنوعة وعدم كفاءة طرق التكيف الحالية.

La introducción de SegMoTE, un marco de mezcla de expertos a nivel de token para la segmentación de imágenes médicas, aborda desafíos significativos en el campo, particularmente las limitaciones de modelos existentes como SAM para adaptarse a diversas modalidades de imagen médica y la ineficiencia de los métodos de adaptación actuales.

L'introduction de SegMoTE, un cadre de mélange d'experts au niveau des tokens pour la segmentation d'images médicales, répond à des défis significatifs dans le domaine, en particulier les limitations des modèles existants comme SAM à s'adapter à diverses modalités d'imagerie médicale et les inefficacités des méthodes d'adaptation actuelles.

The introduction of SegMoTE, a Token-Level Mixture of Experts framework for medical image segmentation, addresses significant challenges in the field, particularly the limitations of existing models like SAM in adapting to diverse medical imaging modalities and the inefficiencies of current adaptation methods.

Segment to Focus: Guiding Latent Action Models in the Presence of Distractors

Was this article worth reading? Share it

Ready to build your own newsroom?