arXiv:2507.11017v2 Announce Type: replace-cross 
Abstract: Post-training quantization (PTQ) offers an efficient approach to compressing large language models (LLMs), significantly reducing memory access and computational costs. Existing compensation-based weight calibration methods often rely on a second-order Taylor expansion to model quantization error, under the assumption that the first-order term is negligible in well-trained full-precision models. However, we reveal that the progressive compensation process introduces accumulated first-order deviations between latent weights and their full-precision counterparts, making this assumption fundamentally flawed. To address this, we propose FOEM, a novel PTQ method that explicitly incorporates first-order gradient terms to improve quantization error compensation. FOEM approximates gradients by performing a first-order Taylor expansion around the pre-quantization weights. This yields an approximation based on the difference between latent and full-precision weights as well as the Hessian matrix. When substituted into the theoretical solution, the formulation eliminates the need to explicitly compute the Hessian, thereby avoiding the high computational cost and limited generalization of backpropagation-based gradient methods. This design introduces only minimal additional computational overhead. Extensive experiments across a wide range of models and benchmarks demonstrate that FOEM consistently outperforms the classical GPTQ method. In 3-bit weight-only quantization, FOEM reduces the perplexity of Llama3-8B by 17.3% and increases the 5-shot MMLU accuracy from 53.8% achieved by GPTAQ to 56.1%. Moreover, FOEM can be seamlessly combined with advanced techniques such as SpinQuant, delivering additional gains under the challenging W4A4KV4 setting and further narrowing the performance gap with full-precision baselines, surpassing existing state-of-the-art methods.

تقدم الكوانتية بعد التدريب (PTQ) طريقة فعالة لضغط نماذج اللغة الكبيرة (LLMs)، مما يؤدي إلى تقليل تكاليف الذاكرة والحساب. غالبًا ما تعتمد طرق معايرة الوزن القائمة على التعويض على توسيع تايلور من الدرجة الثانية لمعالجة خطأ الكوانتية، مع افتراض أن الحدود من الدرجة الأولى غير ذات أهمية. ومع ذلك، تكشف هذه الدراسة أن عملية التعويض يمكن أن تؤدي إلى انحرافات تراكمية من الدرجة الأولى، مما يتحدى هذا الافتراض. تقترح الطريقة الجديدة FOEM تضمين حدود التدرج من الدرجة الأولى لتحسين تعويض خطأ الكوانتية.

La cuantificación post-entrenamiento (PTQ) es un método eficaz para comprimir grandes modelos de lenguaje (LLMs), lo que reduce los costos de memoria y computación. Los métodos de calibración de pesos basados en compensación a menudo utilizan una expansión de Taylor de segundo orden para abordar el error de cuantificación, asumiendo que los términos de primer orden son despreciables. Sin embargo, este estudio revela que el proceso de compensación puede llevar a desviaciones acumuladas de primer orden, desafiando esta suposición. El método propuesto FOEM incorpora términos de gradiente de prime…

La quantification post-formation (PTQ) est une méthode efficace pour compresser les grands modèles de langage (LLMs), entraînant une réduction des coûts de mémoire et de calcul. Les méthodes de calibration des poids basées sur la compensation utilisent souvent une expansion de Taylor d'ordre deux pour traiter l'erreur de quantification, en supposant que les termes d'ordre un sont négligeables. Cependant, cette étude révèle que le processus de compensation peut entraîner des écarts d'ordre un significatifs, remettant en question cette hypothèse. La méthode FOEM proposée intègre des termes de gr…

Post-training quantization (PTQ) is an effective method for compressing large language models (LLMs), leading to reduced memory and computational costs. Traditional weight calibration methods often use a second-order Taylor expansion to address quantization error, assuming first-order terms are negligible. However, this study reveals that the compensation process can lead to significant first-order deviations, challenging this assumption. The proposed FOEM method incorporates first-order gradient terms to enhance quantization error compensation, improving model performance.

First-Order Error Matters: Accurate Compensation for Quantized Large Language Models

Was this article worth reading? Share it

LucidQuery AI

Langfuse

Hypertune

FastML

Sellm

Octofy

Ready to build your own newsroom?