PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs

MORPH has been introduced as a modality-agnostic, autoregressive foundation model designed for partial differential equations (PDEs), utilizing a convolutional vision transformer backbone to manage diverse spatiotemporal datasets across various resolutions and data modalities. The model incorporates advanced techniques such as component-wise convolution and inter-field cross-attention to enhance its predictive capabilities.

Recent advancements in fine-tuning methodologies have led to the introduction of IniLoRA, a novel initialization strategy designed to optimize Low-Rank Adaptation (LoRA) for large language models. IniLoRA initializes low-rank matrices to closely approximate original model weights, addressing limitations in performance seen with traditional LoRA methods. Experimental results demonstrate that IniLoRA outperforms LoRA across various models and tasks, with two additional variants, IniLoRA-$\alpha$ and IniLoRA-$\beta$, further enhancing performance.

A novel method called Dual LoRA has been proposed to enhance the performance of Low-Rank Adaptation (LoRA) in fine-tuning large language models (LLMs). This method introduces two distinct groups within low-rank matrices: a magnitude group for controlling the extent of parameter updates and a direction group for determining the update direction, thereby improving the adaptation process.

The introduction of NAS-LoRA represents a significant advancement in the adaptation of the Segment Anything Model (SAM) for specialized tasks, particularly in medical and agricultural imaging. This new Parameter-Efficient Fine-Tuning (PEFT) method integrates a Neural Architecture Search (NAS) block to enhance SAM's performance by addressing its limitations in acquiring high-level semantic information due to the lack of spatial priors in its Transformer encoder.

A recent study highlights the vulnerabilities of proactive defenses against deepfakes, revealing that these defenses often lack the necessary robustness and reliability. The research introduces a novel technique called Low-Rank Adaptation (LoRA) patching, which effectively bypasses existing defenses by injecting adaptable patches into deepfake generators. This method also includes a Multi-Modal Feature Alignment loss to ensure semantic consistency in outputs.

Delta Sampling (DS) has been introduced as a novel method for enabling data-free knowledge transfer across different diffusion models, particularly addressing the challenges faced when upgrading base models like Stable Diffusion. This method operates at inference time, utilizing the delta between model predictions before and after adaptation, thus facilitating the reuse of adaptation components across varying architectures.

Recent advancements in diffusion models have led to the development of a phase-aware strategy that accelerates image generation by applying different speedups to various stages of the process. This approach utilizes lightweight LoRA adapters, named Slow-LoRA and Fast-LoRA, to enhance efficiency without extensive retraining of models.

A recent empirical survey examined seven model merging algorithms aimed at mitigating social bias in large language models (LLMs), including Linear, Karcher Mean, and SLERP, among others. The study evaluated their effectiveness using 13 open weight models from the GPT, LLaMA, and Qwen families against three bias datasets: BBQ, BOLD, and HONEST, while also assessing their impact on downstream performance in tasks from the SuperGLUE benchmark.