ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation

The study presents a novel Mixture-of-Ranks (MoR) architecture for real-world image super-resolution (Real-ISR), integrating sparse Mixture-of-Experts (MoE) into existing frameworks. This approach aims to enhance the adaptability of models in capturing the diverse characteristics of degraded images while facilitating knowledge sharing among inputs. The proposed method utilizes a fine-grained expert partitioning strategy, treating each rank in Low-Rank Adaptation (LoRA) as an independent expert.

The study introduces 'Erase to Retain', a framework for selectively unlearning knowledge in medical segmentation networks. This method allows for targeted forgetting of specific representations without the need for complete retraining, utilizing a teacher-student distillation approach combined with Low-Rank Adaptation (LoRA). The framework enhances privacy compliance and ethical deployment in medical imaging by enabling the erasure of sensitive information while maintaining overall anatomical understanding.

Federated learning (FL) allows clients to collaboratively train a shared model in a distributed manner, differing from traditional deep learning. This research introduces a new open-source framework for benchmarking FL models under dynamic client participation (DPFL), addressing the challenges of clients intermittently joining or leaving during training. The framework offers configurable data distributions and evaluation metrics, revealing significant performance degradation in FL models under DPFL conditions.

LoRA on the Go (LoGo) introduces a training-free framework for dynamic selection and merging of Low-Rank Adaptation (LoRA) adapters at the instance level. This approach addresses the limitations of conventional LoRA adapters, which are typically trained for single tasks. By leveraging signals from a single forward pass, LoGo identifies the most relevant adapters for diverse tasks, enhancing performance across multiple NLP benchmarks without the need for additional labeled data.

HAWAII is a proposed framework aimed at enhancing the efficiency of vision-language models (VLMs) by distilling knowledge from multiple visual experts into a single vision encoder. This approach minimizes computational costs while retaining the strengths of various experts. The framework employs teacher-specific Low-Rank Adaptation (LoRA) adapters to manage knowledge transfer effectively, reducing conflicts and improving performance in visual understanding tasks.