arXiv:2511.15515v1 Announce Type: new 
Abstract: Recent CLIP-based few-shot semantic segmentation methods introduce class-level textual priors to assist segmentation by typically using a single prompt (e.g., a photo of class). However, these approaches often result in incomplete activation of target regions, as a single textual description cannot fully capture the semantic diversity of complex categories. Moreover, they lack explicit cross-modal interaction and are vulnerable to noisy support features, further degrading visual prior quality. To address these issues, we propose the Multi-Text Guided Few-Shot Semantic Segmentation Network (MTGNet), a dual-branch framework that enhances segmentation performance by fusing diverse textual prompts to refine textual priors and guide the cross-modal optimization of visual priors. Specifically, we design a Multi-Textual Prior Refinement (MTPR) module that suppresses interference and aggregates complementary semantic cues to enhance foreground activation and expand semantic coverage for structurally complex objects. We introduce a Text Anchor Feature Fusion (TAFF) module, which leverages multi-text embeddings as semantic anchors to facilitate the transfer of discriminative local prototypes from support images to query images, thereby improving semantic consistency and alleviating intra-class variations. Furthermore, a Foreground Confidence-Weighted Attention (FCWA) module is presented to enhance visual prior robustness by leveraging internal self-similarity within support foreground features. It adaptively down-weights inconsistent regions and effectively suppresses interference in the query segmentation process. Extensive experiments on standard FSS benchmarks validate the effectiveness of MTGNet. In the 1-shot setting, it achieves 76.8% mIoU on PASCAL-5i and 57.4% on COCO-20i, with notable improvements in folds exhibiting high intra-class variations.

أبرزت التطورات الأخيرة في تقسيم الدلالات القليلة القائمة على CLIP القيود في التقاط التنوع الدلالي للفئات المعقدة. يقترح شبكة تقسيم الدلالات القليلة الموجهة بواسطة نصوص متعددة (MTGNet) معالجة هذه المشكلات من خلال استخدام إطار عمل مزدوج الفرع يدمج عدة مطالبات نصية، مما يعزز أداء التقسيم من خلال تحسين الأولويات النصية وتحسين التفاعل بين الأنماط البصرية.

Los recientes avances en la segmentación semántica de pocos disparos utilizando métodos basados en CLIP han destacado las limitaciones para capturar la diversidad semántica de categorías complejas. La propuesta de la Red de Segmentación Semántica Guiada por Múltiples Textos (MTGNet) aborda estos problemas mediante un marco de doble rama que integra múltiples indicaciones textuales, mejorando el rendimiento de la segmentación a través de prioridades textuales refinadas y una mejor optimización cruzada de prioridades visuales.

Des avancées récentes dans la segmentation sémantique à quelques exemples utilisant des méthodes basées sur CLIP ont mis en évidence des limitations dans la capture de la diversité sémantique des catégories complexes. Le réseau de segmentation sémantique à quelques exemples guidé par plusieurs textes (MTGNet) propose de remédier à ces problèmes en utilisant un cadre à double branche qui intègre plusieurs invites textuelles, améliorant ainsi la performance de segmentation grâce à des priorités textuelles affinées et une optimisation croisée des priorités visuelles.

Recent advancements in few-shot semantic segmentation using CLIP-based methods have highlighted limitations in capturing the semantic diversity of complex categories. The proposed Multi-Text Guided Few-Shot Semantic Segmentation Network (MTGNet) addresses these issues by employing a dual-branch framework that integrates multiple textual prompts, enhancing segmentation performance through refined textual priors and improved cross-modal optimization of visual priors.

Multi-Text Guided Few-Shot Semantic Segmentation

arXiv:2504.15669v3 Announce Type: replace 
Abstract: Few-shot semantic segmentation has attracted growing interest for its ability to generalize to novel object categories using only a few annotated samples. To address data scarcity, recent methods incorporate multiple foundation models to improve feature transferability and segmentation performance. However, they often rely on dual-branch architectures that combine pre-trained encoders to leverage complementary strengths, a design that limits flexibility and efficiency. This raises a fundamental question: can we build a unified model that integrates knowledge from different foundation architectures? Achieving this is, however, challenging due to the misalignment between class-agnostic segmentation capabilities and fine-grained discriminative representations. To this end, we present UINO-FSS, a novel framework built on the key observation that early-stage DINOv2 features exhibit distribution consistency with SAM's output embeddings. This consistency enables the integration of both models' knowledge into a single-encoder architecture via coarse-to-fine multimodal distillation. In particular, our segmenter consists of three core components: a bottleneck adapter for embedding alignment, a meta-visual prompt generator that leverages dense similarity volumes and semantic embeddings, and a mask decoder. Using hierarchical cross-model distillation, we effectively transfer SAM's knowledge into the segmenter, further enhanced by Mamba-based 4D correlation mining on support-query pairs. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ show that UINO-FSS achieves new state-of-the-art results under the 1-shot setting, with mIoU of 80.6 (+3.8%) on PASCAL-5$^i$ and 64.5 (+4.1%) on COCO-20$^i$, demonstrating the effectiveness of our unified approach.

UINO-FSS هو إطار جديد مصمم لتجزئة المعاني القليلة، مما يسمح بالتعميم على فئات جديدة من الكائنات باستخدام عدد قليل فقط من العينات المعلّمة. يتناول نقص البيانات من خلال دمج المعرفة من نماذج أساسية متعددة، متجاوزًا قيود الهياكل ذات الفرعين. يستفيد الإطار من ميزات DINOv2 في المراحل المبكرة التي تتماشى جيدًا مع تمثيلات الإخراج لنموذج Segment Anything (SAM).

UINO-FSS es un nuevo marco diseñado para la segmentación semántica de pocos disparos, que permite la generalización a nuevas categorías de objetos utilizando solo unas pocas muestras anotadas. Aborda la escasez de datos al integrar conocimientos de varios modelos fundamentales, superando las limitaciones de las arquitecturas de doble rama. El marco aprovecha las características tempranas de DINOv2 que se alinean bien con las incrustaciones de salida del Segment Anything Model (SAM).

UINO-FSS est un nouveau cadre conçu pour la segmentation sémantique à faible échantillonnage, permettant la généralisation à de nouvelles catégories d'objets avec un minimum d'échantillons annotés. Il répond à la rareté des données en intégrant des connaissances provenant de divers modèles fondamentaux, surmontant les limitations des architectures à double branche. Le cadre exploite les caractéristiques précoces de DINOv2 qui s'alignent bien avec les embeddings de sortie du Segment Anything Model (SAM).

UINO-FSS is a novel framework designed for few-shot semantic segmentation, which allows for generalization to new object categories using minimal annotated samples. It addresses data scarcity by integrating knowledge from various foundation models, overcoming limitations of dual-branch architectures. The framework leverages early-stage DINOv2 features that align well with the output embeddings of the Segment Anything Model (SAM).

Multi-Text Guided Few-Shot Semantic Segmentation

Was this article worth reading? Share it