arXiv:2511.12281v1 Announce Type: cross 
Abstract: Motivated by the high costs of using black-box Large Language Models (LLMs), we introduce a novel prompt compression paradigm, under which we use smaller LLMs to compress inputs for the larger ones. We present the first comprehensive LLM-as-a-compressor benchmark spanning 25 open- and closed-source models, which reveals significant disparity in models' compression ability in terms of (i) preserving semantically important information (ii) following the user-provided compression rate (CR). We further improve the performance of gpt-4.1-mini, the best overall vanilla compressor, with Textgrad-based compression meta-prompt optimization. We also identify the most promising open-source vanilla LLM - Qwen3-4B - and post-train it with a combination of supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), pursuing the dual objective of CR adherence and maximizing the downstream task performance. We call the resulting model Cmprsr and demonstrate its superiority over both extractive and vanilla abstractive compression across the entire range of compression rates on lengthy inputs from MeetingBank and LongBench as well as short prompts from GSM8k. The latter highlights Cmprsr's generalizability across varying input lengths and domains. Moreover, Cmprsr closely follows the requested compression rate, offering fine control over the cost-quality trade-off.

يقدم المقال Cmprsr، وهو نموذج جديد لضغط المطالبات يهدف إلى تقليل التكاليف المرتبطة باستخدام نماذج اللغة الكبيرة (LLMs) ذات الصندوق الأسود. من خلال استخدام نماذج لغة أصغر لضغط المدخلات للنماذج الأكبر، يقدم البحث معيارًا عبر 25 نموذجًا، مما يبرز الفروقات الكبيرة في قدرتها على الحفاظ على المعلومات المهمة والامتثال لمعدلات الضغط المحددة من قبل المستخدم. تم تحسين أداء gpt-4.1-mini من خلال تقنيات تحسين، بينما تم تحديد Qwen3-4B كنموذج مفتوح المصدر واعد تم تدريبه بعد ذلك لتحسين الأداء في المهام.

El artículo presenta Cmprsr, un nuevo paradigma de compresión de prompts que busca reducir los costos asociados con el uso de grandes modelos de lenguaje (LLM) de caja negra. Al utilizar modelos de lenguaje más pequeños para comprimir las entradas destinadas a modelos más grandes, el estudio presenta un banco de pruebas que abarca 25 modelos, destacando diferencias significativas en su capacidad para preservar información importante y cumplir con las tasas de compresión definidas por el usuario. Se mejora el rendimiento de gpt-4.1-mini mediante técnicas de optimización, mientras que Qwen3-4B s…

L'article présente Cmprsr, un nouveau paradigme de compression de prompts visant à réduire les coûts associés à l'utilisation de grands modèles de langage (LLM) en boîte noire. En utilisant de plus petits LLM pour compresser les entrées destinées à des modèles plus grands, l'étude présente une évaluation de 25 modèles, mettant en évidence des différences significatives dans leur capacité à préserver des informations importantes et à respecter les taux de compression définis par l'utilisateur. La performance de gpt-4.1-mini est améliorée grâce à des techniques d'optimisation, tandis que Qwen3-4…

The article introduces Cmprsr, a novel prompt compression paradigm aimed at reducing the costs associated with using large black-box language models (LLMs). By utilizing smaller LLMs to compress inputs for larger ones, the study presents a benchmark across 25 models, highlighting significant differences in their ability to preserve important information and adhere to user-defined compression rates. The performance of gpt-4.1-mini is enhanced through optimization techniques, while Qwen3-4B is identified as a promising open-source model that undergoes post-training to improve task performance.

Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor

arXiv:2511.17562v1 Announce Type: new 
Abstract: This paper introduces ChineseErrorCorrector3-4B, a unified model for Chinese spelling and grammatical error correction based on Qwen3-4B. The model demonstrates outstanding performance in general text correction tasks and achieves state-of-the-art results in both spelling correction (CSC) and grammatical correction (CGC). On several authoritative benchmark datasets -- including SIGHAN-2015, EC-LAW, MCSC, and NaCGEC -- the model's F1 and F0.5 scores significantly surpass existing publicly available models, ranking first in both spelling and grammatical error correction tasks.

يمثل تقديم ChineseErrorCorrector3-4B تقدمًا كبيرًا في مجال الذكاء الاصطناعي، وخاصة في تصحيح الأخطاء الإملائية والنحوية باللغة الصينية. لقد أظهر هذا النموذج الموحد، المستند إلى Qwen3-4B، أداءً استثنائيًا عبر مجموعة متنوعة من مجموعات البيانات المرجعية، بما في ذلك SIGHAN-2015 وEC-LAW، محققًا أعلى الدرجات في مهام تصحيح الأخطاء الإملائية والنحوية.

La introducción de ChineseErrorCorrector3-4B marca un avance significativo en el campo de la inteligencia artificial, específicamente en la corrección de errores ortográficos y gramaticales en chino. Este modelo unificado, basado en Qwen3-4B, ha demostrado un rendimiento excepcional en varios conjuntos de datos de referencia, incluidos SIGHAN-2015 y EC-LAW, logrando las mejores puntuaciones en tareas de corrección ortográfica y gramatical.

L'introduction de ChineseErrorCorrector3-4B représente une avancée significative dans le domaine de l'intelligence artificielle, en particulier dans la correction des erreurs d'orthographe et de grammaire en chinois. Ce modèle unifié, basé sur Qwen3-4B, a démontré des performances exceptionnelles sur divers ensembles de données de référence, y compris SIGHAN-2015 et EC-LAW, atteignant des scores de premier plan dans les tâches de correction orthographique et grammaticale.

The introduction of ChineseErrorCorrector3-4B marks a significant advancement in the field of artificial intelligence, specifically in the area of Chinese spelling and grammatical error correction. This unified model, based on Qwen3-4B, has demonstrated exceptional performance across various benchmark datasets, including SIGHAN-2015 and EC-LAW, achieving top scores in both spelling and grammatical correction tasks.

ChineseErrorCorrector3-4B: State-of-the-Art Chinese Spelling and Grammar Corrector

arXiv:2508.06447v2 Announce Type: replace 
Abstract: Long-context inference for Large Language Models (LLMs) is heavily limited by high computational demands. While several existing methods optimize attention computation, they still process the full set of hidden states at each layer, limiting overall efficiency. In this work, we propose SlimInfer, an innovative framework that aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass. Our key insight is an information diffusion phenomenon: As information from critical tokens propagates through layers, it becomes distributed across the entire sequence. This diffusion process suggests that LLMs can maintain their semantic integrity when excessive tokens, even including these critical ones, are pruned in hidden states. Motivated by this, SlimInfer introduces a dynamic fine-grained pruning mechanism that accurately removes redundant tokens of hidden state at intermediate layers. This layer-wise pruning naturally enables an asynchronous KV cache manager that prefetches required token blocks without complex predictors, reducing both memory usage and I/O costs. Extensive experiments show that SlimInfer can achieve up to $\mathbf{2.53\times}$ time-to-first-token (TTFT) speedup and $\mathbf{1.88\times}$ end-to-end latency reduction for LLaMA3.1-8B-Instruct on a single RTX 4090, without sacrificing performance on LongBench. Our code is available at https://github.com/Longxmas/SlimInfer.

تم تقديم SlimInfer كإطار مصمم لتحسين كفاءة الاستدلال طويل السياق في نماذج اللغة الكبيرة (LLMs) من خلال تنفيذ تقليم ديناميكي للتوكنات. تتيح هذه الطريقة المبتكرة إزالة التوكنات الأقل أهمية أثناء عملية التمرير للأمام، مما يحسن الموارد الحاسوبية مع الحفاظ على السلامة الدلالية.

SlimInfer se ha presentado como un marco diseñado para mejorar la eficiencia de la inferencia de largo contexto en los Modelos de Lenguaje de Gran Tamaño (LLMs) mediante la implementación de la poda dinámica de tokens. Este enfoque innovador permite la eliminación de tokens menos críticos durante el paso hacia adelante, optimizando los recursos computacionales mientras se mantiene la integridad semántica.

SlimInfer a été introduit comme un cadre conçu pour améliorer l'efficacité de l'inférence à long contexte dans les modèles de langage de grande taille (LLMs) en mettant en œuvre une taille dynamique des tokens. Cette approche innovante permet de supprimer les tokens moins critiques pendant le passage avant, optimisant ainsi les ressources informatiques tout en maintenant l'intégrité sémantique.

SlimInfer has been introduced as a framework designed to enhance the efficiency of long-context inference in Large Language Models (LLMs) by implementing dynamic token pruning. This innovative approach allows for the removal of less critical tokens during the forward pass, optimizing computational resources while maintaining semantic integrity.

SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning

arXiv:2511.17127v1 Announce Type: new 
Abstract: We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs with Pollara interconnect. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts on Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.

تم إجراء دراسة تدريب على نطاق واسع باستخدام مزيج من الخبراء (MoE) باستخدام أجهزة AMD النقية، وبالتحديد وحدات معالجة الرسوميات MI300X مع الاتصال Pollara. توفر هذه الدراسة إرشادات عملية حول تصميم الأنظمة والنماذج، بما في ذلك قياسات دقيقة شاملة لجمع البيانات الأساسية وقياسات MI300X الدقيقة لحجم النواة وعرض النطاق الترددي للذاكرة.

Se ha llevado a cabo un estudio de preentrenamiento a gran escala sobre una mezcla de expertos (MoE) utilizando hardware puro de AMD, específicamente GPUs MI300X con interconexión Pollara. Este estudio proporciona orientación práctica sobre el diseño de sistemas y modelos, incluidos microbenchmarks completos para colectivos centrales y microbenchmarks MI300X para el tamaño de núcleos y el ancho de banda de memoria.

Une étude de préentraînement à grande échelle sur un mélange d'experts (MoE) a été réalisée en utilisant du matériel AMD pur, en particulier des GPU MI300X avec interconnexion Pollara. Cette étude fournit des conseils pratiques sur la conception des systèmes et des modèles, y compris des microbenchmarks complets pour les collectifs de base et des microbenchmarks MI300X pour la taille des noyaux et la bande passante mémoire.

A large-scale mixture-of-experts (MoE) pretraining study has been conducted using pure AMD hardware, specifically MI300X GPUs with Pollara interconnect. This study provides practical guidance on system and model design, including comprehensive microbenchmarks for core collectives and MI300X microbenchmarks for kernel sizing and memory bandwidth.

Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor

Was this article worth reading? Share it

Augmeta

Agentcloud

Prompt Builder