arXiv:2510.27391v2 Announce Type: replace-cross 
Abstract: Modality alignment is critical for vision-language models (VLMs) to effectively integrate information across modalities. However, existing methods extract hierarchical features from text while representing each image with a single feature, leading to asymmetric and suboptimal alignment. To address this, we propose Alignment across Trees, a method that constructs and aligns tree-like hierarchical features for both image and text modalities. Specifically, we introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers, guided by textual cues to extract visual features with coarse-to-fine semantics. We then embed the feature trees of the two modalities into hyperbolic manifolds with distinct curvatures to effectively model their hierarchical structures. To align across the heterogeneous hyperbolic manifolds with different curvatures, we formulate a KL distance measure between distributions on heterogeneous manifolds, and learn an intermediate manifold for manifold alignment by minimizing the distance. We prove the existence and uniqueness of the optimal intermediate manifold. Experiments on taxonomic open-set classification tasks across multiple image datasets demonstrate that our method consistently outperforms strong baselines under few-shot and cross-domain settings.

تم اقتراح طريقة جديدة تُدعى Alignment across Trees لتحسين توافق الأنماط في نماذج الرؤية واللغة (VLMs) من خلال بناء وتوافق ميزات هرمية على شكل شجرة لكل من أنماط الصورة والنص. تستخدم هذه الطريقة إطارًا لاستخراج الميزات البصرية الواعية بالمعنى الذي يستخدم آليات الانتباه المتبادل، مما يسمح بدمج المعلومات بشكل أكثر فعالية عبر الأنماط المختلفة.

Se ha propuesto un nuevo método llamado Alignment across Trees para mejorar la alineación de modalidades en los modelos de visión-lenguaje (VLMs) mediante la construcción y alineación de características jerárquicas en forma de árbol para las modalidades de imagen y texto. Este enfoque utiliza un marco de extracción de características visuales consciente del significado que emplea mecanismos de atención cruzada, permitiendo una integración más efectiva de la información a través de diferentes modalidades.

Une nouvelle méthode appelée Alignment across Trees a été proposée pour améliorer l'alignement des modalités dans les modèles de vision-langage (VLMs) en construisant et en alignant des caractéristiques hiérarchiques en forme d'arbre pour les modalités d'image et de texte. Cette approche utilise un cadre d'extraction de caractéristiques visuelles conscient du sémantique qui emploie des mécanismes d'attention croisée, permettant une intégration plus efficace des informations à travers différentes modalités.

A new method called Alignment across Trees has been proposed to enhance modality alignment in vision-language models (VLMs) by constructing and aligning tree-like hierarchical features for both image and text modalities. This approach utilizes a semantic-aware visual feature extraction framework that employs cross-attention mechanisms, enabling a more effective integration of information across different modalities.