MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
The recent introduction of MoBiQuant, a novel Mixture-of-Bits quantization framework, addresses the challenges of deploying large language models (LLMs) with varying quantization precisions, enhancing their adaptability to dynamic computational resources. This framework aims to resolve issues related to poor generalization in existing post-training quantization methods, which struggle with precision-dependent outlier migration.
WPN Brief
- What Happened
The recent introduction of MoBiQuant, a novel Mixture-of-Bits quantization framework, addresses the challenges of deploying large language models (LLMs) with varying quantization precisions, enhancing their adaptability to dynamic computational resources. This framework aims to resolve issues related to poor generalization in existing post-training quantization methods, which struggle with precision-dependent outlier migration.
- Why It Matters
This development is significant as it allows for more efficient LLM inference, potentially improving performance in real-time applications where computational resources may fluctuate. By enabling models to adjust weight precision dynamically, MoBiQuant could lead to broader adoption of LLMs in various sectors, including AI-driven applications and services.
- The Bigger Picture
The advancement of MoBiQuant aligns with ongoing efforts in the AI community to enhance model efficiency and adaptability. This trend is reflected in various approaches to optimizing LLMs, such as Next Implicit Token Prediction and cooperative memory paging, which also seek to improve the functionality and usability of language models in diverse contexts.
Related Reports
More coverage on this story
10 reports across the wire
Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission
arXiv:2505.11788v2 Announce Type: replace-cross Abstract: To support emerging language-based applications using dispersed and heterogeneous computing resources, the hybrid language model (HLM) offers a promising architecture, where an on-device small language model (SLM) generates draft tokens that are validated and corrected by a remote large language model (LLM). However, the original HLM suffers from substantial communication overhead, as the LLM requires the SLM to upload the full vocabulary distribution for each token. Moreover, both communication and computation resources are wasted when the LLM validates tokens that are highly likely to be accepted. To overcome these limitations, we propose communication-efficient and uncertainty-aware HLM (CU-HLM). In CU-HLM, the SLM transmits truncated vocabulary distributions only when its output uncertainty is high. We validate the feasibility of this opportunistic transmission by discovering a strong correlation between SLM's uncertainty and LLM's rejection probability. Furthermore, we theoretically derive optimal uncertainty thresholds and optimal vocabulary truncation strategies. Simulation results show that, compared to standard HLM, CU-HLM achieves up to 206$\times$ higher token throughput by skipping 74.8% transmissions with 97.4% vocabulary compression, while maintaining 97.4% accuracy.
Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models
A recent study introduced Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models (dMLLMs), which enhances decoding by addressing visual redundancy in token selection. This method utilizes a Visual Redundancy Index (VRI) to optimize the selection of tokens at multiple masked positions, ensuring that high-confidence tokens do not rely on overlapping visual grounding.
Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning
arXiv:2605.24058v1 Announce Type: new Abstract: On-device adaptation of large language models commonly keeps a quantized base model frozen while training and deploying a small, task-specific LoRA adapter. In the unmerged adapter-mode setting, however, the adapter is more than a compact storage module; it introduces an additional dense floating-point branch, maintains a trainable state for local updates, and acts as a unit of communication and hot-swapping.We introduce LoRDBA, a LoRA-compatible adapter that replaces both low-rank factors with binary sign carriers while representing magnitudes through lightweight, channel-wise scales, converting the dense adapter branch into two sign-accumulation matrix multiplications interleaved with channel-wise scaling. A finite-sample analysis shows that reconstruction quality is governed by the residual-to-magnitude ratio of the original LoRA factors. In adapter-mode experiments, LoRDBA outperforms low-bit baselines at matched model sizes while matching fp16 LoRA quality in selected regimes. The unmerged adapter incurs at most 8% prefill latency overhead at matched rank r=16 despite an over 10x reduction in adapter footprint, with moderate training memory overhead of approximately 1.6x that of fp16 LoRA.
How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
arXiv:2605.25698v1 Announce Type: new Abstract: High-quality data is scarce in large language model (LLM) training, yet how to schedule its use jointly with training dynamics lacks theoretical guidance. We extend functional scaling laws by incorporating a data-quality dimension, and solve the joint data-quality and batch-size scheduling problem in asymptotic closed form. The solution reveals two regimes and a dual role of high-quality data. In the noise-limited regime, high-quality data should be used as a signal amplifier: lowering the batch size converts cleaner data into more signal without amplifying noise. In the signal-limited regime, it should be used as a noise suppressor: late placement reduces terminal noise without sacrificing signal accumulation. Existing curriculum-style pipelines primarily exploit the second role by placing cleaner data late, but miss the first role because conventional decay schedules reduce update intensity exactly when high-quality data becomes available. Guided by this, we propose Drop-Stable-Rampup for LLM midtraining: upon the quality transition, drop the batch size, hold it stable to accumulate signal, then ramp up to suppress terminal noise. On a 15B Mixture-of-Experts model midtrained on 108B tokens, Drop-Stable-Rampup improves average accuracy over Warmup-Stable-Decay (WSD) by +1.70 and over Cosine-decay by +2.98, with particularly large gains on mathematical reasoning benchmarks such as GSM8K (+4.23) and MATH (+2.80).
NITP: Next Implicit Token Prediction for LLM Pre-training
The introduction of Next Implicit Token Prediction (NITP) aims to enhance the pre-training of large language models (LLMs) by addressing limitations in standard next-token prediction methods. NITP incorporates dense continuous supervision in the representation space, allowing models to predict the implicit semantic content of the next token, thereby improving generalization and representation structure.
JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogates
arXiv:2605.25469v1 Announce Type: new Abstract: Quantization-aware training (QAT) is widely deployed but typically relies on the Straight-Through Estimator (STE), which passes gradients through non-differentiable quantizers by fiat. This often makes training brittle near bin boundaries and weakly aligned with the actual behavior of the low-precision model. We introduce JacQuant, a QAT framework that learns a lightweight surrogate of the model's local sensitivity to parameter changes and uses it to stabilize and accelerate training within standard variance-reduced optimizers. The surrogate is inexpensive (diagonal or block-diagonal), data-driven, and compatible with common weight and activation quantizers. On code-preserving training phases, we prove convergence for non-convex objectives and obtain linear rates under a PL condition, and we relate the learned sensitivity to end-to-end output fidelity via a simple calibration argument. Across LLM benchmarks at $\leq 2$ bits, JacQuant consistently reaches higher accuracy than STE-based QAT, and the runtime analyses on various models show that the added cost remains negligible under practical group sizes. The method is drop-in and requires no changes to the forward quantizers; our empirical claims are scoped to ultra-low-bit LLM QAT.
Large Language Model Selection with Limited Annotations
A new framework called SELECT-LLM has been introduced to facilitate the selection of Large Language Models (LLMs) for specific tasks by identifying a minimal set of queries that yield the most informative annotations. This approach leverages expected information gain based on pairwise similarities between model outputs, making it applicable to both open-weight and black-box models.
LLMs Show No Signs Of Individuated Metacognition
arXiv:2605.24299v1 Announce Type: new Abstract: Confidence-weighted routing, selective abstention, and ensemble weighting all assume that a model's stated confidence is informative about its capability on the question being asked. They presume functional metacognition, the capacity to assess one's own capabilities, without exercising them. Aggregate calibration is well studied, with mixed results, but the underlying structure of elicited confidence is less well understood. We decompose binary confidence judgements from 20 frontier Large Language Models (LLMs) across six benchmarks using tetrachoric factor analysis paired with pairwise calibration, asking whether two models that differ in confidence also differ in performance. On factual recall and information retrieval benchmarks the cross-model confidence matrix is approximately rank-one and a single dominant factor captures most of the latent variance. Models retrieving facts share an item-level difficulty axis and differ mainly in their decision thresholds along it. Across all benchmarks the relationship between confidence and performance collapses once items that all models agree on are removed. Inter-model pairwise calibration is small even where statistically significant, and what remains shrinks to nothing once base-rate differences along the shared factor are controlled for. Mathematical reasoning is the apparent exception, but this turns out to be a confound where reasoning models answer questions about their confidence by trying to solve them in their chain of thought, bypassing the sub-symbolic self-knowledge we seek to measure. We find no evidence for significant verbalised individuated metacognition in any tested domain.
Cooperative Memory Paging with Keyword Bookmarks for Long-Horizon LLM Conversations
A new approach called cooperative paging has been proposed for managing long-horizon conversations in large language models (LLMs), allowing models to replace evicted content with keyword bookmarks and retrieve full content on demand. This method has shown superior performance on the LoCoMo benchmark, outperforming traditional retrieval methods across multiple LLMs.
The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models
A recent study introduces TraceLock, a lightweight plug-in controller designed to learn a token-commitment policy for diffusion language models, addressing the challenge of selecting which proposed tokens to incorporate into partially decoded sequences. This approach aims to enhance the efficiency of token generation by leveraging self-supervision derived from future stability during the decoding process.