arXiv:2410.17770v3 Announce Type: replace 
Abstract: This work analyzes singular-value spectra of weight matrices in pretrained transformer models to understand how information is stored at both ends of the spectrum. Using Random Matrix Theory (RMT) as a zero information hypothesis, we associate agreement with RMT as evidence of randomness and deviations as evidence for learning. Surprisingly, we observe pronounced departures from RMT not only among the largest singular values -- the usual outliers -- but also among the smallest ones. A comparison of the associated singular vectors with the eigenvectors of the activation covariance matrices shows that there is considerable overlap wherever RMT is violated. Thus, significant directions in the data are captured by small singular values and their vectors as well as by the large ones. We confirm this empirically: zeroing out the singular values that deviate from RMT raises language-model perplexity far more than removing values from the bulk, and after fine-tuning the smallest decile can be the third most influential part of the spectrum. To explain how vectors linked to small singular values can carry more information than those linked to larger values, we propose a linear random-matrix model. Our findings highlight the overlooked importance of the low end of the spectrum and provide theoretical and practical guidance for SVD-based pruning and compression of large language models.

تتناول دراسة حديثة طيف القيم الفردية لمصفوفات الأوزان في نماذج المحولات المدربة مسبقًا، كاشفةً كيف يتم تخزين المعلومات في هذه الأنظمة المعقدة. من خلال تطبيق نظرية المصفوفات العشوائية، وجد الباحثون انحرافات ملحوظة عن الأنماط المتوقعة، مما يشير إلى أن هذه النماذج ليست عشوائية فحسب، بل تعلمت تمثيلات ذات مغزى. هذه الرؤية مهمة لأنها تعزز فهمنا لكيفية عمل نماذج المحولات، مما قد يؤدي إلى تحسينات في تصميمها وتطبيقها في مجالات متنوعة.

Un estudio reciente analiza los espectros de valores singulares de las matrices de pesos en modelos de transformadores preentrenados, revelando cómo se almacena la información en estos sistemas complejos. Al aplicar la Teoría de Matrices Aleatorias, los investigadores encontraron desviaciones significativas de los patrones esperados, lo que indica que estos modelos no son solo aleatorios, sino que han aprendido representaciones significativas. Este conocimiento es crucial, ya que mejora nuestra comprensión de cómo funcionan los modelos de transformadores, lo que podría llevar a mejoras en su diseño y aplicación en diversos campos.

Une étude récente examine les spectres des valeurs singulières des matrices de poids dans les modèles de transformateurs préentraînés, révélant comment l'information est stockée dans ces systèmes complexes. En appliquant la théorie des matrices aléatoires, les chercheurs ont trouvé des écarts significatifs par rapport aux modèles attendus, indiquant que ces modèles ne sont pas simplement aléatoires mais ont appris des représentations significatives. Cette compréhension est cruciale car elle améliore notre connaissance du fonctionnement des modèles de transformateurs, ce qui pourrait conduire à des améliorations dans leur conception et leur application dans divers domaines.

A recent study delves into the singular-value spectra of weight matrices in pretrained transformer models, revealing how information is stored within these complex systems. By applying Random Matrix Theory, the researchers found significant deviations from expected patterns, indicating that these models are not just random but have learned meaningful representations. This insight is crucial as it enhances our understanding of how transformer models function, potentially leading to improvements in their design and application in various fields.

Small Singular Values Matter: A Random Matrix Analysis of Transformer Models

Was this article worth reading? Share it

LucidQuery AI

The Visualizer

VibeFrame

AIPortalX

One More Thing in AI

Hypertune

Ready to build your own newsroom?