arXiv:2605.26399v1 Announce Type: new 
Abstract: Understanding human gaze behavior is essential for complex scene comprehension and human-computer interaction. Traditional gaze following models are typically restricted to pure spatial localization, lacking the high-level capacity to reason about semantic targets or complex social contexts. Furthermore, these models often process individuals sequentially, requiring redundant computations over the same scene image for multi-person inference. While recent Vision-Language Models (VLMs) offer the exceptional semantic reasoning needed to address gaze-related semantic tasks, their reliance on discrete text generation inherently limits precision in continuous spatial tasks like gaze localization. To bridge this gap, we propose OmniGF, a unified vision-language framework that adapts foundational VLMs for highly scalable multi-person gaze reasoning. The model adopts a dual-branch decoding strategy: a structured language branch generates discrete reasoning states, while a continuous spatial branch directly taps into the VLM's dense hidden states. Supervising these extracted representations with high-resolution gaze target heatmaps effectively overcomes the spatial bottleneck of text-only coordinate generation. Furthermore, to explicitly ground the model in multi-person scenes, we augment the input with head embeddings encoded from cropped head images, providing fine-grained appearance and orientation cues for all individuals simultaneously. By modeling all individuals and leveraging the strong semantic capability of VLMs, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning. Extensive experiments demonstrate that our framework establishes new state-of-the-art performance across multiple standard benchmarks. Code is available at https://github.com/cvlab-stonybrook/omnigf.

تم تقديم OmniGF، وهو إطار عمل للرؤية واللغة ذو فرعين، بهدف تعزيز قدرات متابعة النظر من خلال دمج التفكير الدلالي عالي المستوى مع تحديد المواقع، مما يعالج القيود الموجودة في النماذج التقليدية التي تعالج الأفراد بشكل متسلسل.

La introducción de OmniGF, un marco de visión-lenguaje de doble rama, tiene como objetivo mejorar las capacidades de seguimiento de la mirada al integrar el razonamiento semántico de alto nivel con la localización espacial, abordando las limitaciones de los modelos tradicionales que procesan a los individuos de manera secuencial.

L'introduction d'OmniGF, un cadre de vision-langage à double branche, vise à améliorer les capacités de suivi du regard en intégrant un raisonnement sémantique de haut niveau avec la localisation spatiale, répondant aux limites des modèles traditionnels qui traitent les individus de manière séquentielle.

The introduction of OmniGF, a dual-branch vision-language framework, aims to enhance gaze following capabilities by integrating high-level semantic reasoning with spatial localization, addressing limitations in traditional models that process individuals sequentially.