arXiv:2605.27465v1 Announce Type: new 
Abstract: The quadratic cost of self-attention in Vision Transformers (ViTs) constitutes a fundamental bottleneck for practical deployment, motivating a vibrant line of research on token reduction. Among existing approaches, token merging (ToMe) has emerged as an elegant training-free solution; yet its design rests on an unspoken premise of token equality, which contravenes the well-documented non-uniformity of self-attention and leads to information loss in high-salience tokens under aggressive compression. We address this limitation with AdaMerge, a token-merging framework based on two complementary mechanisms. First, salience-weighted similarity leverages column-wise feature-affinity centrality as a token-importance proxy and incorporates the resulting salience scores into the bipartite matching score, ensuring that pivotal tokens contribute more strongly to the merged representation. Second, adaptive merging intensity uses pre-computed layer-wise similarity statistics to dynamically modulate the per-layer reduction count in accordance with input-specific redundancy. On ImageNet-1k with ViT-B/16, AdaMerge consistently outperforms ToMe, PiToMe, and DSM across all FLOPs-matched regimes. The accuracy gap widens monotonically with compression: at the 13.4G FLOPs operating point, AdaMerge sustains a Top-1 degradation of only -1.06%, compared to -1.45% for PiToMe and -4.62% for DSM. To our knowledge, AdaMerge is the first to combine salience-weighted similarity and adaptive per-layer reduction into a single training-free token merging framework, advancing the accuracy-FLOPs Pareto frontier of ViT acceleration.

تم تقديم AdaMerge كإطار عمل لدمج الرموز التكيفية والواعية للأهمية، بهدف تسريع المحولات البصرية (ViTs) دون الحاجة إلى التدريب. تتناول هذه الطريقة المبتكرة قيود طرق دمج الرموز الحالية من خلال التركيز على أهمية الرموز ذات الأهمية العالية، مما يقلل من فقدان المعلومات أثناء الضغط العدواني.

AdaMerge ha sido presentado como un marco de fusión de tokens adaptativo y consciente de la saliencia, con el objetivo de acelerar los Transformers de Visión (ViTs) sin necesidad de entrenamiento. Este enfoque innovador aborda las limitaciones de los métodos de fusión de tokens existentes al centrarse en la importancia de los tokens de alta saliencia, reduciendo así la pérdida de información durante la compresión agresiva.

AdaMerge a été introduit comme un cadre de fusion de tokens adaptatif et conscient de la saillance, visant à accélérer les Transformers de Vision (ViTs) sans nécessiter d'entraînement. Cette approche innovante s'attaque aux limitations des méthodes de fusion de tokens existantes en se concentrant sur l'importance des tokens à haute saillance, réduisant ainsi la perte d'information lors de la compression agressive.

AdaMerge has been introduced as a salience-aware adaptive token merging framework aimed at accelerating Vision Transformers (ViTs) without the need for training. This innovative approach addresses the limitations of existing token merging methods by focusing on the importance of high-salience tokens, thereby reducing information loss during aggressive compression.