arXiv:2504.15669v4 Announce Type: replace 
Abstract: Few-shot semantic segmentation has attracted growing interest for its ability to generalize to novel object categories using only a few annotated samples. To address data scarcity, recent methods incorporate multiple foundation models to improve feature transferability and segmentation performance. However, they often rely on dual-branch architectures that combine pre-trained encoders to leverage complementary strengths, a design that limits flexibility and efficiency. This raises a fundamental question: can we build a unified model that integrates knowledge from different foundation architectures? Achieving this is, however, challenging due to the misalignment between class-agnostic segmentation capabilities and fine-grained discriminative representations. To this end, we present UINO-FSS, a novel framework built on the key observation that early-stage DINOv2 features exhibit distribution consistency with SAM's output embeddings. This consistency enables the integration of both models' knowledge into a single-encoder architecture via coarse-to-fine multimodal distillation. In particular, our segmenter consists of three core components: a bottleneck adapter for embedding alignment, a meta-visual prompt generator that leverages dense similarity volumes and semantic embeddings, and a mask decoder. Using hierarchical cross-model distillation, we effectively transfer SAM's knowledge into the segmenter, further enhanced by Mamba-based 4D correlation mining on support-query pairs. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ show that UINO-FSS achieves new state-of-the-art results under the 1-shot setting, with mIoU of 80.6 (+3.8%) on PASCAL-5$^i$ and 64.5 (+4.1%) on COCO-20$^i$, demonstrating the effectiveness of our unified approach.

UINO-FSS هو إطار جديد لتجزئة الدلالات القليلة الذي يدمج المعرفة من نماذج أساسية متعددة. يعالج القيود المفروضة على الهياكل ذات الفرعين من خلال الاستفادة من التناسق بين ميزات DINOv2 المبكرة ونتائج SAM. يعزز هذا النهج قابلية نقل الميزات وأداء التجزئة، مما يسهل التعميم على فئات جديدة من الكائنات باستخدام عدد قليل من العينات المعلّمة.

UINO-FSS es un nuevo marco para la segmentación semántica de pocos disparos que integra el conocimiento de varios modelos de base. Aborda las limitaciones de las arquitecturas de doble rama al aprovechar la consistencia entre las características tempranas de DINOv2 y los embeddings de salida de SAM. Este enfoque mejora la transferibilidad de características y el rendimiento de segmentación, facilitando la generalización a nuevas categorías de objetos con muestras anotadas mínimas.

UINO-FSS est un nouveau cadre pour la segmentation sémantique à faible échantillonnage qui intègre les connaissances de divers modèles de base. Il aborde les limitations des architectures à double branche en tirant parti de la cohérence entre les caractéristiques précoces de DINOv2 et les embeddings de sortie de SAM. Cette approche améliore la transférabilité des caractéristiques et la performance de segmentation, facilitant la généralisation à de nouvelles catégories d'objets avec un minimum d'échantillons annotés.

UINO-FSS is a novel framework for few-shot semantic segmentation that integrates knowledge from various foundation models. It addresses the limitations of existing dual-branch architectures by leveraging the consistency between early-stage DINOv2 features and SAM's output embeddings. This approach enhances feature transferability and segmentation performance, making it easier to generalize to new object categories with minimal annotated samples.

UINO-FSS: Unifying Representation Learning and Few-shot Segmentation via Hierarchical Distillation and Mamba-HyperCorrelation

arXiv:2507.14137v3 Announce Type: replace 
Abstract: We present Franca (pronounced Fran-ka): free one; the first fully open-source (data, code, weights) vision foundation model that matches and in many cases surpasses the performance of state-of-the-art proprietary models, e.g., DINOv2, CLIP, SigLIPv2, etc. Our approach is grounded in a transparent training pipeline inspired by Web-SSL and uses publicly available data: ImageNet-21K and a subset of ReLAION-2B. Beyond model release, we tackle critical limitations in SSL clustering methods. While modern models rely on assigning image features to large codebooks via clustering algorithms like Sinkhorn-Knopp, they fail to account for the inherent ambiguity in clustering semantics. To address this, we introduce a parameter-efficient, multi-head clustering projector based on nested Matryoshka representations. This design progressively refines features into increasingly fine-grained clusters without increasing the model size, enabling both performance and memory efficiency. Additionally, we propose a novel positional disentanglement strategy that explicitly removes positional biases from dense representations, thereby improving the encoding of semantic content. This leads to consistent gains on several downstream benchmarks, demonstrating the utility of cleaner feature spaces. Our contributions establish a new standard for transparent, high-performance vision models and open a path toward more reproducible and generalizable foundation models for the broader AI community. The code and model checkpoints are available at https://github.com/valeoai/Franca.

تم تقديم فرانكا، أول نموذج مفتوح المصدر بالكامل في مجال الرؤية، والذي يظهر أداءً يعادل أو يتجاوز نماذج الملكية مثل DINOv2 و CLIP. يستخدم هذا النموذج خط أنابيب تدريب شفاف ومجموعات بيانات متاحة للجمهور، مما يعالج القيود في طرق التجميع الحالية في التعلم الذاتي من خلال نهج جديد للتجميع المتداخل على غرار ماتريوشكا.

Franca, el primer modelo de fundación visual completamente de código abierto, ha sido presentado, mostrando un rendimiento que iguala o supera a modelos propietarios como DINOv2 y CLIP. Este modelo utiliza un pipeline de entrenamiento transparente y conjuntos de datos disponibles públicamente, abordando las limitaciones de los métodos de agrupamiento en el aprendizaje auto-supervisado a través de un nuevo enfoque de agrupamiento anidado tipo Matryoshka.

Franca, le premier modèle de fondation visuelle entièrement open-source, a été présenté, affichant des performances qui égalent ou dépassent celles de modèles propriétaires comme DINOv2 et CLIP. Ce modèle utilise un pipeline d'entraînement transparent et des ensembles de données disponibles publiquement, abordant les limitations des méthodes de clustering en apprentissage auto-supervisé grâce à une nouvelle approche de clustering imbriqué de type Matryoshka.

Franca, the first fully open-source vision foundation model, has been introduced, showcasing performance that matches or exceeds proprietary models like DINOv2 and CLIP. This model utilizes a transparent training pipeline and publicly available datasets, addressing limitations in current self-supervised learning clustering methods through a novel nested Matryoshka clustering approach.

Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning

One More Thing in AI – Your Shortcut to AI Mastery

UINO-FSS: Unifying Representation Learning and Few-shot Segmentation via Hierarchical Distillation and Mamba-HyperCorrelation

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Octofy

Https

Metaflow AI

Dyad

Ready to build your own newsroom?