arXiv:2606.03712v1 Announce Type: new 
Abstract: Graph Language Models (GLMs) have become a promising direction for adapting Large Language Models (LLMs) to graph learning tasks. By transforming graph topology and node information into graph tokens, GLMs allow LLMs to jointly process structured graph inputs and textual instructions. Yet, it remains unclear how LLMs internally interpret these graph tokens and whether graph tokens act as meaningful carriers of graph structure. In this work, we analyze how LLMs process graph information through graph-token behavior in representative GLM architectures.
  Findings. We find that the internal saliency of graph tokens in GLMs is not equivalent to graph information utilization. Graph sink tokens consistently emerge as activation-level outliers: they can be identified by massive activation values along a small set of hidden-state dimensions and are biased toward early graph-token positions. However, this activation-level saliency does not imply that these tokens are the main carriers of graph information. Unlike classical attention sinks in language and vision-language models, graph sink tokens do not necessarily attract the largest attention weights from query tokens. Through pruning, repositioning, and swapping interventions, we show that graph sink tokens are not the most important semantic or structural tokens for downstream prediction.
  Implications. Together, these results suggest that after current GLMs map graph structure into the LLM token space, the resulting graph-token representations do not naturally form a fully usable topology-aware internal representation; instead, they exhibit a decoupling between activation-level saliency and graph-semantic utility. This decoupling points to limitations in existing graph-token construction, placement, and alignment mechanisms.

دراسة حديثة بعنوان 'When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models' تستكشف سلوك نماذج اللغة الرسومية (GLMs) في معالجة رموز الرسوم البيانية، كاشفة أن الأهمية الداخلية لهذه الرموز لا تعادل الاستخدام الفعال للمعلومات الرسومية. تبرز الدراسة ظهور رموز الغرق الرسومية كاستثناءات بارزة في مستويات التنشيط.

Un estudio reciente titulado 'When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models' explora el comportamiento de los Graph Language Models (GLMs) en el procesamiento de tokens gráficos, revelando que la saliencia interna de estos tokens no se traduce en una utilización efectiva de la información gráfica. La investigación destaca la aparición de tokens de hundimiento gráfico como valores atípicos significativos en los niveles de activación.

Une étude récente intitulée 'When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models' examine le comportement des Graph Language Models (GLMs) dans le traitement des graph tokens, révélant que la saillance interne de ces tokens ne correspond pas à une utilisation efficace des informations graphiques. La recherche met en lumière l'émergence de graph sink tokens comme des valeurs aberrantes significatives dans les niveaux d'activation.

A recent study titled 'When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models' explores the behavior of Graph Language Models (GLMs) in processing graph tokens, revealing that the internal saliency of these tokens does not equate to effective graph information utilization. The research highlights the emergence of graph sink tokens as significant outliers in activation levels.