arXiv:2409.14980v2 Announce Type: replace 
Abstract: We introduce a (de)-regularization of the Maximum Mean Discrepancy (DrMMD) and its Wasserstein gradient flow. Existing gradient flows that transport samples from source distribution to target distribution with only target samples, either lack tractable numerical implementation ($f$-divergence flows) or require strong assumptions, and modifications such as noise injection, to ensure convergence (Maximum Mean Discrepancy flows). In contrast, DrMMD flow can simultaneously (i) guarantee near-global convergence for a broad class of targets in both continuous and discrete time, and (ii) be implemented in closed form using only samples. The former is achieved by leveraging the connection between the DrMMD and the $\chi^2$-divergence, while the latter comes by treating DrMMD as MMD with a de-regularized kernel. Our numerical scheme uses an adaptive de-regularization schedule throughout the flow to optimally trade off between discretization errors and deviations from the $\chi^2$ regime. The potential application of the DrMMD flow is demonstrated across several numerical experiments, including a large-scale setting of training student/teacher networks.

تم تقديم طريقة جديدة تُسمى (de)-regularized Maximum Mean Discrepancy (DrMMD)، والتي تعزز التدفقات التدرجية الحالية لنقل العينات بين توزيعات المصدر والهدف. تضمن هذه الطريقة تقاربًا شبه عالمي لمجموعة واسعة من الأهداف ويمكن تنفيذها بشكل مغلق باستخدام العينات فقط، مما يعالج القيود الموجودة في الأساليب السابقة مثل تدفقات $f$-divergence وMaximum Mean Discrepancy.

Se ha introducido un nuevo método llamado (de)-regularized Maximum Mean Discrepancy (DrMMD), que mejora los flujos de gradiente existentes para el transporte de muestras entre distribuciones de origen y objetivo. Este método asegura una convergencia casi global para una amplia gama de objetivos y puede implementarse en forma cerrada utilizando solo muestras, abordando las limitaciones encontradas en enfoques anteriores como los flujos de $f$-divergencia y Maximum Mean Discrepancy.

Une nouvelle méthode appelée (de)-regularized Maximum Mean Discrepancy (DrMMD) a été introduite, améliorant les flux de gradient existants pour le transport d'échantillons entre des distributions source et cible. Cette méthode garantit une convergence quasi-globale pour une large gamme de cibles et peut être mise en œuvre sous forme fermée en utilisant uniquement des échantillons, répondant ainsi aux limitations des approches précédentes telles que les flux de $f$-divergence et de Maximum Mean Discrepancy.

A new method called (de)-regularized Maximum Mean Discrepancy (DrMMD) has been introduced, which enhances the existing gradient flows for transporting samples between source and target distributions. This method ensures near-global convergence for a wide range of targets and can be implemented in closed form using only samples, addressing limitations found in previous approaches like $f$-divergence and Maximum Mean Discrepancy flows.

(De)-regularized Maximum Mean Discrepancy Gradient Flow

arXiv:2511.17339v1 Announce Type: new 
Abstract: Prompt learning has emerged as an effective technique for fine-tuning large-scale foundation models for downstream tasks. However, conventional prompt tuning methods are prone to overfitting and can struggle with out-of-distribution generalization. To address these limitations, Bayesian prompt learning has been proposed, which frames prompt optimization as a Bayesian inference problem to enhance robustness. This paper introduces Repulsive Bayesian Prompt Learning (ReBaPL), a novel method for Bayesian prompt learning, designed to efficiently explore the complex and often multimodal posterior landscape of prompts. Our method integrates a cyclical step-size schedule with a stochastic gradient Hamiltonian Monte Carlo (SGHMC) algorithm, enabling alternating phases of exploration to discover new modes, and exploitation to refine existing modes. Furthermore, we introduce a repulsive force derived from a potential function over probability metrics (including Maximum Mean Discrepancy and Wasserstein distance) computed on the distributions of representations produced by different prompts. This representation-space repulsion diversifies exploration and prevents premature collapse to a single mode. Our approach allows for a more comprehensive characterization of the prompt posterior distribution, leading to improved generalization. In contrast to prior Bayesian prompt learning methods, our method provides a modular plug-and-play Bayesian extension of any existing prompt learning method based on maximum likelihood estimation. We demonstrate the efficacy of ReBaPL on several benchmark datasets, showing superior performance over state-of-the-art methods for prompt learning.

تم تقديم طريقة جديدة تُدعى التعلم البايزي الطارد للمطالبات (ReBaPL) لتحسين تحسين المطالبات في نماذج الأساس على نطاق واسع. تتناول هذه الطريقة القيود المفروضة على طرق ضبط المطالبات التقليدية، التي غالبًا ما تعاني من الإفراط في التكيف وصعوبة التعميم خارج التوزيع من خلال تأطير تحسين المطالبات كمشكلة استدلال بايزي.

Se ha introducido un nuevo método llamado Repulsive Bayesian Prompt Learning (ReBaPL) para mejorar la optimización de prompts en modelos de fundación a gran escala. Este enfoque aborda las limitaciones de los métodos de ajuste de prompts convencionales, que a menudo luchan con el sobreajuste y la generalización fuera de distribución al enmarcar la optimización de prompts como un problema de inferencia bayesiana.

Une nouvelle méthode appelée Repulsive Bayesian Prompt Learning (ReBaPL) a été introduite pour améliorer l'optimisation des invites dans les modèles de fondation à grande échelle. Cette approche répond aux limitations des méthodes d'optimisation d'invites conventionnelles, qui luttent souvent contre le surajustement et la généralisation hors distribution en formulant l'optimisation des invites comme un problème d'inférence bayésienne.

A new method called Repulsive Bayesian Prompt Learning (ReBaPL) has been introduced to enhance prompt optimization in large-scale foundation models. This approach addresses the limitations of conventional prompt tuning methods, which often struggle with overfitting and out-of-distribution generalization by framing prompt optimization as a Bayesian inference problem.

ReBaPL: Repulsive Bayesian Prompt Learning

arXiv:2308.01853v3 Announce Type: replace 
Abstract: Contaminations are a key concern in modern statistical learning, as small but systematic perturbations of all datapoints can substantially alter estimation results. Here, we study Wasserstein-$r$ contaminations ($r\ge 1$) in an $\ell_q$ norm ($q\in [1,\infty]$), in which each observation may undergo an adversarial perturbation with bounded cost, complementing the classical Huber model, corresponding to total variation norm, where only a fraction of observations is arbitrarily corrupted. We study both independent and joint (coordinated) contaminations and develop a minimax theory under $\ell_q^r$ losses.
  Our analysis encompasses several fundamental problems: location estimation, linear regression, and pointwise nonparametric density estimation. For joint contaminations in location estimation and for prediction in linear regression, we obtain the exact minimax risk, identify least favorable contaminations, and show that the sample mean and least squares predictor are respectively minimax optimal. For location estimation under independent contaminations, we give sharp upper and lower bounds, including exact minimaxity in the Euclidean Wasserstein contamination case, when $q=r=2$. For pointwise density estimation in any dimension, we derive the optimal rate, showing that it is achieved by kernel density estimation with a bandwidth that is possibly larger than the classical one.
  Our proofs leverage powerful tools from optimal transport developed over the last 20 years, including the dynamic Benamou-Brenier formulation. Taken together, our results suggest that in contrast to the Huber contamination model, for norm-based Wasserstein contaminations, classical estimators may be nearly optimally robust.

قدمت دراسة حديثة إطارًا لتقدير إحصائي مصغر تحت تلوث فاسرشتاين، حيث تتناول الاضطرابات النظامية في البيانات التي يمكن أن تؤثر بشكل كبير على نتائج التقدير. تستكشف هذه البحث تلوثات فاسرشتاين-$r$ في معيار $	ext{l}_q$، موسعة نموذج هوبر الكلاسيكي من خلال النظر في التلوثات المستقلة والمشتركة عبر مشاكل إحصائية متنوعة مثل تقدير الموقع والانحدار الخطي.

Un estudio reciente ha introducido un marco de estimación estadística minimax bajo contaminación de Wasserstein, abordando las perturbaciones sistemáticas en los datos que pueden impactar significativamente los resultados de estimación. Esta investigación explora las contaminaciones Wasserstein-$r$ en una norma $	ext{l}_q$, ampliando el modelo clásico de Huber al considerar tanto contaminaciones independientes como conjuntas en diversos problemas estadísticos como la estimación de ubicación y la regresión lineal.

Une étude récente a introduit un cadre d'estimation statistique minimax sous contamination de Wasserstein, abordant les perturbations systématiques dans les données qui peuvent avoir un impact significatif sur les résultats d'estimation. Cette recherche explore les contaminations Wasserstein-$r$ dans une norme $	ext{l}_q$, étendant le modèle classique de Huber en considérant à la fois des contaminations indépendantes et conjointes à travers divers problèmes statistiques tels que l'estimation de localisation et la régression linéaire.

A recent study has introduced a minimax statistical estimation framework under Wasserstein contamination, addressing systematic perturbations in data that can significantly impact estimation results. This research explores Wasserstein-$r$ contaminations in an $	ext{l}_q$ norm, extending the classical Huber model by considering both independent and joint contaminations across various statistical problems such as location estimation and linear regression.

(De)-regularized Maximum Mean Discrepancy Gradient Flow

Was this article worth reading? Share it

Dublin Maths

Metaflow AI

Dynamiq