arXiv:2511.17826v1 Announce Type: cross 
Abstract: Deterministic inference is increasingly critical for large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and Reinforcement Learning (RL). However, existing LLM serving frameworks exhibit non-deterministic behavior: identical inputs can yield different outputs when system configurations (e.g., tensor parallel (TP) size, batch size) vary, even under greedy decoding. This arises from the non-associativity of floating-point arithmetic and inconsistent reduction orders across GPUs. While prior work has addressed batch-size-related nondeterminism through batch-invariant kernels, determinism across different TP sizes remains an open problem, particularly in RL settings, where the training engine typically uses Fully Sharded Data Parallel (i.e., TP = 1) while the rollout engine relies on multi-GPU TP to maximize the inference throughput, creating a natural mismatch between the two. This precision mismatch problem may lead to suboptimal performance or even collapse for RL training. We identify and analyze the root causes of TP-induced inconsistency and propose Tree-Based Invariant Kernels (TBIK), a set of TP-invariant matrix multiplication and reduction primitives that guarantee bit-wise identical results regardless of TP size. Our key insight is to align intra- and inter-GPU reduction orders through a unified hierarchical binary tree structure. We implement these kernels in Triton and integrate them into vLLM and FSDP. Experiments confirm zero probability divergence and bit-wise reproducibility for deterministic inference across different TP sizes. Also, we achieve bit-wise identical results between vLLM and FSDP in RL training pipelines with different parallel strategy. Code is available at https://github.com/nanomaoli/llm_reproducibility.

تم تسليط الضوء على التقدم الأخير في الاستدلال الحتمي لنماذج اللغة الكبيرة (LLMs)، حيث تم تناول المشكلة الحرجة المتمثلة في عدم التوافق بين التدريب والاستدلال الناجم عن اختلاف أحجام التوازي التنسوري وتكوينات الدفعات. يمكن أن تؤدي هذه المشكلة إلى نتائج غير متسقة، خاصة في تطبيقات التعلم المعزز حيث يتم استخدام تكوينات مختلفة للتدريب والاستدلال.

Se han destacado avances recientes en la inferencia determinista para modelos de lenguaje de gran tamaño (LLMs), abordando el problema crítico de la descoordinación entre el entrenamiento y la inferencia que surge de las variaciones en los tamaños de paralelismo tensorial y las configuraciones de lotes. Esta descoordinación puede llevar a resultados inconsistentes, especialmente en aplicaciones de aprendizaje por refuerzo donde se emplean diferentes configuraciones para el entrenamiento y la inferencia.

Des avancées récentes dans l'inférence déterministe pour les modèles de langage de grande taille (LLMs) ont été mises en avant, abordant le problème critique de l'inadéquation entre l'entraînement et l'inférence qui découle de la variation des tailles de parallélisme tensoriel et des configurations de lot. Cette inadéquation peut entraîner des résultats incohérents, en particulier dans les applications d'apprentissage par renforcement où différentes configurations sont utilisées pour l'entraînement et l'inférence.

Recent advancements in deterministic inference for large language models (LLMs) have been highlighted, addressing the critical issue of training-inference mismatch that arises from varying tensor parallel sizes and batch configurations. This mismatch can lead to inconsistent outputs, particularly in reinforcement learning applications where different configurations are employed for training and inference.

Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch

Was this article worth reading? Share it

Hypertune

One More Thing in AI

LCW