arXiv:2510.21501v1 Announce Type: new 
Abstract: Vision encoders are indispensable for allowing impressive performance of Multi-modal Large Language Models (MLLMs) in vision language tasks such as visual question answering and reasoning. However, existing vision encoders focus on global image representations but overlook fine-grained regional analysis. They are limited in fine grained perception due to the scarcity of fine grained annotated data and the lack of a fine grained pre-training paradigm. In this paper, we propose GranViT, a novel Vision Transformer that integrates fine-grained feature extraction with semantic alignment to Large Language Models (LLMs) via region level autoregressive training. We first construct Gran-29M, a dataset comprising 2million natural and OCR images paired with over 180 million high-quality region-level annotations, to enable large scale fine grained pretraining. Consequently, we develop a pretraining-adaptation framework along with a self distillation mechanism to train fine-grained GranViT on Gran-29M. We sufficiently exploit the fine-grained annotations from Gran-29M to resort to bounding-box-to-caption regression to enhance localized visual representation of the vision encoder in the pretraining and caption-to-bounding-box regression to improve vision feature utilization and localization for LLM in the adaptation. We further incorporate a self distillation mechanism that imposes explicit localization constraints on the vision encoder to strengthen its regional reasoning capability. Extensive experiments show that GranViT surpasses existing vision encoders and attains strong transferability to varying LLMs. Remarkably, it achieves state-of-the-art results on fine-grained recognition, multimodal VQA, and OCR understanding.

GranViT هو نموذج رؤية مبتكر يعزز قدرات نماذج اللغة متعددة الوسائط (MLLM) من خلال التركيز على الإدراك الدقيق. على عكس مشفرات الرؤية التقليدية التي تحلل بشكل أساسي الميزات العامة للصور، يركز GranViT على التحليل الإقليمي التفصيلي، وهو أمر حاسم لمهام مثل الإجابة على الأسئلة المرئية. هذه الابتكار مهم لأنه يعالج قيود النماذج الحالية التي تكافح مع البيانات الدقيقة بسبب نقص الموارد المعلّمة. من خلال تحسين كيفية فهم الآلات للصور، يمكن أن يؤدي GranViT إلى تفاعلات أكثر دقة وعمقًا في تطبيقات الذكاء الاصطناعي.

GranViT es un modelo de visión innovador que mejora las capacidades de los Modelos de Lenguaje Multimodal (MLLM) al centrarse en la percepción de detalles finos. A diferencia de los codificadores de visión tradicionales que analizan principalmente características globales de las imágenes, GranViT enfatiza el análisis regional detallado, crucial para tareas como la respuesta a preguntas visuales. Esta innovación es significativa ya que aborda las limitaciones de los modelos existentes que luchan con datos finos debido a la escasez de recursos anotados. Al mejorar cómo las máquinas comprenden las imágenes, GranViT podría llevar a interacciones más precisas y matizadas en aplicaciones de IA.

GranViT est un modèle de vision révolutionnaire qui améliore les capacités des modèles de langage multimodal (MLLM) en se concentrant sur la perception fine. Contrairement aux encodeurs de vision traditionnels qui analysent principalement les caractéristiques globales des images, GranViT met l'accent sur l'analyse régionale détaillée, essentielle pour des tâches comme le questionnement visuel. Cette innovation est importante car elle répond aux limites des modèles existants qui ont du mal avec les données fines en raison d'un manque de ressources annotées. En améliorant la compréhension des images par les machines, GranViT pourrait conduire à des interactions plus précises et nuancées dans les applications d'IA.

GranViT is a groundbreaking vision model that enhances the capabilities of Multi-modal Large Language Models (MLLMs) by focusing on fine-grained perception. Unlike traditional vision encoders that primarily analyze global image features, GranViT emphasizes detailed regional analysis, which is crucial for tasks like visual question answering. This innovation is significant as it addresses the limitations of existing models that struggle with fine-grained data due to a lack of annotated resources. By improving how machines understand images, GranViT could lead to more accurate and nuanced interactions in AI applications.

GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs

One More Thing in AI – Your Shortcut to AI Mastery

GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

The Visualizer

Attentive AI

OpenL Translator

Blunge

Ready to build your own newsroom?