arXiv:2605.16470v1 Announce Type: new 
Abstract: Adapting large language models (LLMs) to downstream tasks via full fine-tuning is increasingly impractical due to its computational and memory demands. Parameter-efficient fine-tuning (PEFT) approaches such as Low-Rank Adaptation (LoRA) mitigate this by confining updates to a compact set of trainable parameters, but this aggressive reduction often sacrifices generalization, especially under transfer across heterogeneous tasks and domains. We revisit the tension between parameter efficiency and adaptation capacity, and ask whether the two are truly at odds. We answer in the negative by introducing LoRA-Over, a framework grounded in a simple principle: enrich the optimization landscape during training, then collapse the enrichment at inference. LoRA-Over injects auxiliary parameters into the low-rank adapters during training to broaden the effective hypothesis space, and through a decomposition-based reformulation folds them back into a standard low-rank structure with negligible reconstruction error, keeping inference cost identical to vanilla LoRA. Since not all weight matrices benefit equally from added capacity, we further propose two scheduling strategies, one statically predefined and one dynamically determined at runtime, that direct extra capacity where most needed. We evaluate LoRA-Over on language understanding (GLUE, T5-Base), dialogue (MT-Bench), arithmetic reasoning (GSM8K), and code generation (HumanEval), using LLaMA 2-7B and LLaMA 3.1-8B. Across all benchmarks and scales, LoRA-Over consistently outperforms vanilla LoRA, showing that principled over-parameterization designed to vanish at inference is an effective lever for improving PEFT generalization. Code will be released upon acceptance.

تم تقديم إطار عمل جديد يسمى LoRA-Over لتحسين قابلية التكيف لنماذج اللغة الكبيرة (LLMs) لمهام مختلفة بعد التدريب، مع معالجة القيود التي تفرضها طرق الضبط الدقيق التقليدية. تعمل هذه الطريقة على إثراء مشهد التحسين أثناء التدريب وتقلصه أثناء الاستدلال، مما يسمح بتحسين التعميم عبر المهام والمجالات المتنوعة.

Se ha introducido un nuevo marco llamado LoRA-Over para mejorar la adaptabilidad de los grandes modelos de lenguaje (LLMs) a diversas tareas posteriores, abordando las limitaciones de los métodos de ajuste fino tradicionales. Este enfoque enriquece el paisaje de optimización durante el entrenamiento y lo colapsa durante la inferencia, permitiendo una mejor generalización a través de tareas y dominios heterogéneos.

Un nouveau cadre appelé LoRA-Over a été introduit pour améliorer l'adaptabilité des grands modèles de langage (LLMs) pour diverses tâches en aval tout en abordant les limitations des méthodes de réglage fin traditionnelles. Cette approche enrichit le paysage d'optimisation pendant l'entraînement et le réduit lors de l'inférence, permettant une meilleure généralisation à travers des tâches et des domaines hétérogènes.

A new framework called LoRA-Over has been introduced to enhance the adaptability of large language models (LLMs) for various downstream tasks while addressing the limitations of traditional fine-tuning methods. This approach enriches the optimization landscape during training and collapses it during inference, allowing for improved generalization across heterogeneous tasks and domains.