FedSEA-LLaMA: A Secure, Efficient and Adaptive Federated Splitting Framework for Large Language Models

FedSEA-LLaMA, introduced in a recent paper, presents a novel federated splitting framework designed to enhance the deployment of large language models (LLMs) in federated environments. The framework tackles significant challenges such as securing transmitted vectors, managing high communication overhead due to the auto-regressive nature of LLMs, and the inflexibility of fixed partition points. By incorporating Gaussian noise for secure vector transmission and employing attention-mask compression, FedSEA-LLaMA not only improves data privacy but also accelerates communication efficiency. Experiments conducted on tasks like natural language understanding, summarization, and conversational QA demonstrate that FedSEA-LLaMA maintains performance levels comparable to centralized models. This advancement is crucial as it allows for the effective use of private data, which is essential for enhancing the capabilities of language models.