When the Same Coefficients Reach Different Places: Asymmetric Realizability in Transplanting Tokenizers across Large Language Models

Recent research has identified an asymmetric realizability gap in the transplantation of tokenizers across large language models, specifically focusing on the Gemma-2-2B model. This study reveals that the same coefficient vector can yield different outcomes in donor and base anchor spans, leading to the creation of 'breaker tokens' that behave differently in various contexts.

Understanding this phenomenon is crucial for improving the efficiency and effectiveness of model fine-tuning and deployment, particularly in applications that rely on cross-vocabulary model composition.

The findings highlight ongoing challenges in the interpretability and reliability of large language models, as previous studies have pointed to vulnerabilities and feature competition within these systems. This underscores the need for continued exploration of the mechanisms underlying model behavior and the implications for safety and performance in AI applications.