arXiv:2510.03731v2 Announce Type: replace-cross 
Abstract: The rapid development of parameter-efficient fine-tuning methods has noticeably improved the efficiency of adapting large language models. Among these, LoRA has gained widespread popularity due to its strong balance of effectiveness and parameter efficiency. However, LoRA relies on initializing two low-rank matrices whose product is zero, which limits its ability to effectively activate and leverage the original model weights-creating a potential bottleneck for optimal performance. To address this limitation, we propose \textbf{IniLoRA}, a novel initialization strategy that initializes the low-rank matrices to closely approximate the original model weights. Experimental results indicate that IniLoRA achieves better performance than LoRA across a range of models and tasks. Additionally, we introduce two variants, IniLoRA-$\alpha$ and IniLoRA-$\beta$, both leveraging distinct initialization methods to enhance performance further.

أدت التطورات الأخيرة في منهجيات الضبط الدقيق إلى تقديم IniLoRA، وهي استراتيجية جديدة للت initialization تهدف إلى تحسين التكيف منخفض الرتبة (LoRA) لنماذج اللغة الكبيرة. تقوم IniLoRA بتهيئة المصفوفات منخفضة الرتبة لتقريبها عن كثب من أوزان النموذج الأصلي، مما يعالج القيود في الأداء التي لوحظت مع طرق LoRA التقليدية. تُظهر النتائج التجريبية أن IniLoRA تتفوق على LoRA عبر مجموعة من النماذج والمهام، مع تقديم نسختين إضافيتين، IniLoRA-$\alpha$ وIniLoRA-$\beta$، لتعزيز الأداء بشكل أكبر.

Los avances recientes en las metodologías de ajuste fino han llevado a la introducción de IniLoRA, una nueva estrategia de inicialización diseñada para optimizar la Adaptación de Bajo Rango (LoRA) para grandes modelos de lenguaje. IniLoRA inicializa matrices de bajo rango para aproximarse estrechamente a los pesos del modelo original, abordando las limitaciones de rendimiento observadas con los métodos tradicionales de LoRA. Los resultados experimentales demuestran que IniLoRA supera a LoRA en varios modelos y tareas, con dos variantes adicionales, IniLoRA-$\alpha$ e IniLoRA-$\beta$, que mejor…

Des avancées récentes dans les méthodologies de fine-tuning ont conduit à l'introduction d'IniLoRA, une nouvelle stratégie d'initialisation conçue pour optimiser l'adaptation à faible rang (LoRA) pour les grands modèles de langage. IniLoRA initialise les matrices à faible rang pour approcher de près les poids du modèle original, abordant les limitations de performance observées avec les méthodes LoRA traditionnelles. Les résultats expérimentaux montrent qu'IniLoRA surpasse LoRA dans divers modèles et tâches, avec deux variantes supplémentaires, IniLoRA-$\alpha$ et IniLoRA-$\beta$, qui améliore…

Recent advancements in fine-tuning methodologies have led to the introduction of IniLoRA, a novel initialization strategy designed to optimize Low-Rank Adaptation (LoRA) for large language models. IniLoRA initializes low-rank matrices to closely approximate original model weights, addressing limitations in performance seen with traditional LoRA methods. Experimental results demonstrate that IniLoRA outperforms LoRA across various models and tasks, with two additional variants, IniLoRA-$\alpha$ and IniLoRA-$\beta$, further enhancing performance.

Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation

Was this article worth reading? Share it

LucidQuery AI

Hypertune

Octofy