arXiv:2512.14954v1 Announce Type: cross 
Abstract: Computing next-token likelihood ratios between two language models (LMs) is a standard task in training paradigms such as knowledge distillation. Since this requires both models to share the same probability space, it becomes challenging when the teacher and student LMs use different tokenizers, for instance, when edge-device deployment necessitates a smaller vocabulary size to lower memory overhead. In this work, we address this vocabulary misalignment problem by uncovering an implicit recursive structure in the commonly deployed Byte-Pair Encoding (BPE) algorithm and utilizing it to create a probabilistic framework for cross-tokenizer likelihood scoring. Our method enables sequence likelihood evaluation for vocabularies different from the teacher model native tokenizer, addressing two specific scenarios: when the student vocabulary is a subset of the teacher vocabulary, and the general case where it is arbitrary. In the subset regime, our framework computes exact likelihoods and provides next-token probabilities for sequential sampling with only O(1) model evaluations per token. When used for distillation, this yields up to a 12% reduction in memory footprint for the Qwen2.5-1.5B model while also improving baseline performance up to 4% on the evaluated tasks. For the general case, we introduce a rigorous lossless procedure that leverages BPE recursive structure, complemented by a fast approximation that keeps large-vocabulary settings practical. Applied to distillation for mathematical reasoning, our approach improves GSM8K accuracy by more than 2% over the current state of the art.

تقدم دراسة جديدة خوارزميات لتقييم احتمالية بين المرمزين تهدف إلى حل مشكلات عدم توافق المفردات في تقطير نماذج اللغة، خاصة عندما تستخدم نماذج المعلم والطالب مرمزين مختلفين. تكشف هذه الأبحاث عن هيكل متكرر ضمن خوارزمية تشفير الأزواج البايتية لتسهيل تقييم الاحتمالية عبر مفردات متنوعة.

Un nuevo estudio presenta algoritmos de puntuación de probabilidad entre tokenizadores destinados a resolver problemas de desalineación de vocabulario en la destilación de modelos de lenguaje, especialmente cuando los modelos maestro y estudiante utilizan diferentes tokenizadores. Esta investigación descubre una estructura recursiva en el algoritmo de codificación de pares de bytes para facilitar la evaluación de probabilidad a través de vocabularios variados.

Une nouvelle étude présente des algorithmes de scoring de vraisemblance inter-tokenizer visant à résoudre les problèmes de désalignement de vocabulaire dans la distillation de modèles de langage, en particulier lorsque les modèles enseignants et étudiants utilisent des tokenizers différents. Cette recherche révèle une structure récursive dans l'algorithme de Byte-Pair Encoding pour faciliter l'évaluation de vraisemblance à travers des vocabulaires variés.

A new study presents cross-tokenizer likelihood scoring algorithms aimed at resolving vocabulary misalignment issues in language model distillation, particularly when teacher and student models utilize different tokenizers. This research uncovers a recursive structure in the Byte-Pair Encoding algorithm to facilitate likelihood evaluation across varying vocabularies.

Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation

Useful picks

LucidQuery AI

Airparser

Langtail

Sellm

ClassX

FastML