FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models

The introduction of softpick, a novel drop-in replacement for softmax in transformer attention mechanisms, addresses issues of attention sink and massive activations, achieving a consistent 0% sink rate in experiments with large models. This advancement allows for the production of hidden states with lower kurtosis and sparser attention maps.

The introduction of HiFi-Mamba, a dual-stream Mamba-based architecture, aims to enhance high-fidelity MRI reconstruction from undersampled k-space data by addressing key limitations of existing Mamba variants. The architecture features stacked W-Laplacian and HiFi-Mamba blocks, which separate low- and high-frequency streams to improve image fidelity and detail.

A recent study introduces a novel approach to federated ridge regression, demonstrating that iterative communication between clients and a central server is unnecessary for achieving exact recovery of the centralized solution. By aggregating sufficient statistics from clients in a single transmission, the server can reconstruct the global solution through matrix inversion, significantly reducing communication overhead.

Recent research highlights a new type of attack on Federated Learning (FL) that compromises the fairness of trained models, revealing that controlling just one client can skew performance distributions across various attributes. This raises concerns about the integrity of models in sensitive applications where fairness is critical.

A recent study has assessed the effectiveness of amortized inference in Bayesian statistics, particularly under varying signal-to-noise ratios and distribution shifts. This method leverages deep neural networks to streamline the inference process, allowing for significant computational savings compared to traditional Bayesian approaches that require extensive likelihood evaluations.