arXiv:2505.11355v2 Announce Type: replace 
Abstract: Gaussian processes (GPs) have gained popularity as flexible machine learning models for regression and function approximation with an in-built method for uncertainty quantification. However, GPs suffer when the amount of training data is large or when the underlying function contains multi-scale features that are difficult to represent by a stationary kernel. To address the former, training of GPs with large-scale data is often performed through inducing point approximations, also known as sparse GP regression (GPR), where the size of the covariance matrices in GPR is reduced considerably through a greedy search on the data set. To aid the latter, deep GPs have gained traction as hierarchical models that resolve multi-scale features by combining multiple GPs. Posterior inference in deep GPs requires a sampling or, more usual, a variational approximation. Variational approximations lead to large-scale stochastic, non-convex optimisation problems and the resulting approximation tends to represent uncertainty incorrectly. In this work, we combine variational learning with MCMC to develop a particle-based expectation-maximisation method to simultaneously find inducing points within the large-scale data (variationally) and accurately train the deep GPs (sampling-based). The result is a highly efficient and accurate methodology for deep GP training on large-scale data. We test our method on standard benchmark problems.

تم استكشاف تقنيات نادرة للانحدار في العمليات الغاوسية العميقة (GPs) لتحسين قابلية التوسع والكفاءة لهذه النماذج، خاصة عند التعامل مع مجموعات بيانات كبيرة أو وظائف متعددة المقاييس معقدة. تبرز الأبحاث استخدام تقريبات النقاط المحورية في الانحدار GPR النادر ومزايا GPs العميقة للنمذجة الهرمية.

Se han explorado técnicas dispersas para la regresión en procesos gaussianos profundos (GPs) para mejorar la escalabilidad y eficiencia de estos modelos, especialmente al tratar con grandes conjuntos de datos o funciones complejas de múltiples escalas. La investigación destaca el uso de aproximaciones por puntos de inducción en la regresión GPR dispersa y las ventajas de los GPs profundos para la modelización jerárquica.

Des techniques éparses pour la régression dans les processus gaussiens profonds (GPs) ont été explorées pour améliorer la scalabilité et l'efficacité de ces modèles, en particulier lors du traitement de grands ensembles de données ou de fonctions multi-échelles complexes. La recherche met en avant l'utilisation d'approximations par points d'induction dans la régression GPR éparse et les avantages des GPs profonds pour la modélisation hiérarchique.

Sparse techniques for regression in deep Gaussian processes (GPs) have been explored to enhance the scalability and efficiency of these models, particularly when dealing with large datasets or complex multi-scale functions. The research highlights the use of inducing point approximations in sparse GP regression (GPR) and the advantages of deep GPs for hierarchical modeling.

Sparse Techniques for Regression in Deep Gaussian Processes

arXiv:2511.19628v1 Announce Type: new 
Abstract: This study investigates the limitations of applying Markov Chain Monte Carlo (MCMC) methods to arbitrary objective functions, focusing on a two-block MCMC framework which alternates between Metropolis-Hastings and Gibbs sampling. While such approaches are often considered advantageous for enabling data-driven regularization, we show that their performance critically depends on the sharpness of the employed likelihood form. By introducing a sharpness parameter and exploring alternative likelihood formulations proportional to the target objective function, we demonstrate how likelihood curvature governs both in-sample performance and the degree of regularization inferred by the training data. Empirical applications are conducted on reinforcement learning tasks: including a navigation problem and the game of tic-tac-toe. The study concludes with a separate analysis examining the implications of extreme likelihood sharpness on arbitrary objective functions stemming from the classic game of blackjack, where the first block of the two-block MCMC framework is replaced with an iterative optimization step. The resulting hybrid approach achieves performance nearly identical to the original MCMC framework, indicating that excessive likelihood sharpness effectively collapses posterior mass onto a single dominant mode.

دراسة حديثة تستكشف قيود تطبيق أساليب سلسلة ماركوف مونت كارلو (MCMC) على الدوال الهدفية العشوائية، مع التركيز على إطار عمل MCMC ثنائي الكتل الذي يتناوب بين أسلوب ميتروبوليس-هاستينغز وأخذ العينات من غيبس. تسلط الدراسة الضوء على أن أداء هذه الأساليب يعتمد بشكل كبير على حدة شكل الاحتمالية المستخدم، حيث تم تقديم معلمة حدة لاستكشاف آثارها على التنظيم والأداء في العينة.

Un estudio reciente investiga las limitaciones de aplicar métodos de Monte Carlo por cadenas de Markov (MCMC) a funciones objetivo arbitrarias, centrándose en un marco MCMC de dos bloques que alterna entre Metropolis-Hastings y muestreo de Gibbs. La investigación destaca que el rendimiento de estos métodos depende en gran medida de la nitidez de la forma de verosimilitud utilizada, introduciendo un parámetro de nitidez para explorar sus efectos en la regularización y el rendimiento en muestra.

Une étude récente examine les limites de l'application des méthodes de Monte Carlo par chaîne de Markov (MCMC) à des fonctions objectives arbitraires, en se concentrant particulièrement sur un cadre MCMC à deux blocs qui alterne entre Metropolis-Hastings et l'échantillonnage de Gibbs. La recherche souligne que la performance de ces méthodes est fortement influencée par la netteté de la forme de vraisemblance utilisée, introduisant un paramètre de netteté pour explorer ses effets sur la régularisation et la performance en échantillon.

A recent study investigates the limitations of applying Markov Chain Monte Carlo (MCMC) methods to arbitrary objective functions, particularly through a two-block MCMC framework that alternates between Metropolis-Hastings and Gibbs sampling. The research highlights that the performance of these methods is significantly influenced by the sharpness of the likelihood form used, introducing a sharpness parameter to explore its effects on regularization and in-sample performance.

Sparse Techniques for Regression in Deep Gaussian Processes

Was this article worth reading? Share it

GPTHuman

GPTHumanizer

Ask On Data