arXiv:2606.13767v1 Announce Type: new 
Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models. However, questions remain about the comparative generalizability of these approaches and how the structural restrictions on low-rank updates preserve effective adaptation performance. We present a historical framing, covering the past (full fine-tuning and original LoRA), the present (different variants of LoRA), and propose simpler, cheaper, parameter-efficient extensions by inducing sparsity within existing LoRA variants: Cheap LoRA (cLA), training a single low-rank factor with the other fixed (deterministically or, in its randomized variant, stochastically), and the chained circulant variant, ${c}^3$LA. We frame cLA as a structured instance of asymmetric LoRA, serving as a controlled column-subspace restriction of full fine-tuning. We derive information-theoretic generalization error bounds for these variants, marking one of the first endeavors in this area. Empirically, we evaluate 11 fine-tuning methods across 10 pre-trained models and 14 datasets, analyzing the fine-tuned models' performance and generalization using tools such as loss landscapes and spectral analysis. Despite the sensitivity of fine-tuned models to the pre-trained model, datasets, and other factors, our study suggests that restricting LoRA-based PEFT methods' adaptation to a sparse, structured column space remains competitive across tasks with their parameter-matched baselines while reducing up to 10% training time and peak GPU memory up to 15%, even with a na\"ive, non-optimized, sparse implementation. Our theoretical and empirical generalization measures provide a more consistent and principled approach to their cost-effective adaptation than commonly used analytical tools. Overview and code are available at: https://elicaden.github.io/Beyond_LoRA/.

أدت التطورات الأخيرة في التكيف منخفض الرتبة (LoRA) إلى تقديم متغيرات جديدة مثل Cheap LoRA (cLA) وLoRA المتسلسلة (${c}^3$LA)، والتي تهدف إلى تحسين كفاءة الذاكرة والحوسبة مع معالجة القضايا المتعلقة بالعمومية. تقترح هذه التطورات امتدادات فعالة من حيث التكلفة وبسيطة من خلال تحفيز التشتت داخل الأطر الحالية لـLoRA.

Los recientes avances en la adaptación de bajo rango (LoRA) han llevado a la introducción de nuevas variantes como Cheap LoRA (cLA) y LoRA encadenada (${c}^3$LA), que buscan mejorar la eficiencia de memoria y computación mientras abordan preocupaciones sobre la generalización. Estos desarrollos proponen extensiones paramétricas más simples y económicas al inducir esparcimiento dentro de los marcos existentes de LoRA.

Des avancées récentes dans l'adaptation à faible rang (LoRA) ont conduit à l'introduction de nouvelles variantes telles que Cheap LoRA (cLA) et LoRA circulaire en chaîne (${c}^3$LA), qui visent à améliorer l'efficacité mémoire et de calcul tout en abordant les préoccupations de généralisabilité. Ces développements proposent des extensions paramétriques plus simples et moins coûteuses en induisant de la sparsité au sein des cadres LoRA existants.

Recent advancements in low-rank adaptation (LoRA) have led to the introduction of new variants such as Cheap LoRA (cLA) and chained circulant LoRA (${c}^3$LA), which aim to enhance memory and compute efficiency while addressing generalizability concerns. These developments propose simpler and cheaper parameter-efficient extensions by inducing sparsity within existing LoRA frameworks.