arXiv:2606.24033v1 Announce Type: new 
Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector. Under RoPE, however, a key's contribution to a future attention logit decomposes into a position-dependent sum over two-dimensional frequency blocks. This makes key-cache quantization a block-wise bit-allocation problem: high-energy RoPE blocks are more sensitive to quantization error and should receive more bits. We introduce Block-GTQ, a RoPE-aware bit allocator for key-cache quantization built on TurboQuant-MSE(TQ-MSE). For each layer and KV head, Block-GTQ computes a label-free energy score for each RoPE block and greedily allocates integer bit widths by marginal gain. Under matched K/V bit budgets, Block-GTQ better preserves RoPE query-key logits on a ten-model diagnostic panel, cutting per-layer MAE by 32-80% at 2 and 3 b/dim K-only quantization and winning all 367/367 layer comparisons against uniform TQ-MSE. These fidelity gains translate to stronger downstream long-context retrieval, understanding, and reasoning. At K2V2 on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task NIAH average from 70.6 to 97.4, and the LongBench-EN average from 36.87 to 53.31. On AIME 2024/2025 with DeepSeek-R1-Distill-Qwen-7B, without an fp16 recent-key buffer, Block-GTQ at K3V2 scores 51.7/37.5, close to fp16's 54.2/37.9, whereas uniform TQ-MSE collapses to 0.0/0.0. We further implement a packed-cache serving path. On a single H800 GPU with Qwen2.5-3B-Instruct, packed K3V3 achieves 3.24x KV-cache compression with fp16-comparable quality, runs 1.34x faster than fp16 FlashAttention2 at 128K context, reduces peak memory from 56.31 GB to 19.85 GB, and remains feasible at 256K and 512K where fp16 OOMs. Code is available at https://github.com/JIA-Lab-research/blockgtq.

تم تقديم طريقة جديدة تُدعى Block-GTQ لتخصيص البتات الواعية لـ RoPE في تقليل حجم ذاكرة التخزين المؤقت للقيم الرئيسية، مما يعالج القيود التي تواجهها الكوانتيزر ذات البت المنخفض الحالية التي تعالج المفاتيح المخزنة كمتجهات مسطحة. تعمل هذه الطريقة على تحسين تخصيص البتات بناءً على حساسية الطاقة لكتل RoPE، مما يحسن بشكل كبير من الحفاظ على سجلات الاستعلام-المفتاح عبر نماذج متعددة.

Se ha introducido un nuevo método llamado Block-GTQ para la asignación de bits consciente de RoPE en la cuantificación de caché clave-valor, abordando las limitaciones de los cuantificadores de bajo bit existentes que tratan las claves en caché como vectores planos. Este método optimiza la asignación de bits según la sensibilidad energética de los bloques RoPE, mejorando significativamente la preservación de los logits de consulta-clave en múltiples modelos.

Une nouvelle méthode appelée Block-GTQ a été introduite pour l'allocation de bits consciente de RoPE dans la quantification du cache clé-valeur, répondant aux limites des quantificateurs à faible bit existants qui traitent les clés mises en cache comme des vecteurs plats. Cette méthode optimise l'allocation de bits en fonction de la sensibilité énergétique des blocs RoPE, améliorant considérablement la préservation des logits de requête-clé à travers plusieurs modèles.

A new method called Block-GTQ has been introduced for RoPE-aware bit allocation in key-value cache quantization, addressing the limitations of existing low-bit quantizers that treat cached keys as flat vectors. This method optimizes bit allocation based on the energy sensitivity of RoPE blocks, significantly improving the preservation of query-key logits across multiple models.