arXiv:2511.17053v1 Announce Type: new 
Abstract: LVLMs have been shown to perform excellently in image-level tasks such as VQA and caption. However, in many instance-level tasks, such as visual grounding and object detection, LVLMs still show performance gaps compared to previous expert models. Meanwhile, although pedestrian tracking is a classical task, there have been a number of new topics in combining object tracking and natural language, such as Referring MOT, Cross-view Referring MOT, and Semantic MOT. These tasks emphasize that models should understand the tracked object at an advanced semantic level, which is exactly where LVLMs excel. In this paper, we propose a new unified Pedestrian Tracking framework, namely OmniPT, which can track, track based on reference and generate semantic understanding of tracked objects interactively. We address two issues: how to model the tracking task into a task that foundation models can perform, and how to make the model output formatted answers. To this end, we implement a training phase consisting of RL-Mid Training-SFT-RL. Based on the pre-trained weights of the LVLM, we first perform a simple RL phase to enable the model to output fixed and supervisable bounding box format. Subsequently, we conduct a mid-training phase using a large number of pedestrian-related datasets. Finally, we perform supervised fine-tuning on several pedestrian tracking datasets, and then carry out another RL phase to improve the model's tracking performance and enhance its ability to follow instructions. We conduct experiments on tracking benchmarks and the experimental results demonstrate that the proposed method can perform better than the previous methods.

تقديم OmniPT، إطار موحد جديد لتتبع المشاة، يستفيد من قدرات نماذج اللغة البصرية الكبيرة (LVLM) لتعزيز تتبع وفهم الكائنات من خلال معالجة دلالية متقدمة. يعالج هذا الإطار الفجوات الحالية في الأداء في المهام على مستوى الكائن، مثل التثبيت البصري واكتشاف الكائنات، والتي كانت تقليديًا تحت سيطرة نماذج الخبراء.

La introducción de OmniPT, un nuevo marco unificado para el seguimiento de peatones, aprovecha las capacidades de los Modelos de Lenguaje Visual de Gran Escala (LVLM) para mejorar el seguimiento y la comprensión de objetos a través de un procesamiento semántico avanzado. Este marco aborda las brechas de rendimiento existentes en tareas a nivel de instancia, como la anclaje visual y la detección de objetos, que tradicionalmente han sido dominadas por modelos expertos.

L'introduction d'OmniPT, un nouveau cadre unifié pour le suivi des piétons, exploite les capacités des grands modèles de langage visuel (LVLM) pour améliorer le suivi et la compréhension des objets grâce à un traitement sémantique avancé. Ce cadre aborde les lacunes de performance existantes dans des tâches de niveau instance telles que le ancrage visuel et la détection d'objets, qui ont traditionnellement été dominées par des modèles experts.

The introduction of OmniPT, a new unified framework for pedestrian tracking, leverages the capabilities of Large Vision Language Models (LVLMs) to enhance object tracking and understanding through advanced semantic processing. This framework addresses existing performance gaps in instance-level tasks like visual grounding and object detection, which have traditionally been dominated by expert models.

OmniPT: Unleashing the Potential of Large Vision Language Models for Pedestrian Tracking and Understanding

One More Thing in AI – Your Shortcut to AI Mastery

OmniPT: Unleashing the Potential of Large Vision Language Models for Pedestrian Tracking and Understanding

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

OpenL Translator

Attentive AI

Sellm

LangWatch

Ready to build your own newsroom?