arXiv:2606.05296v1 Announce Type: new 
Abstract: LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents. Code is available at https://github.com/layer6ai-labs/Agentic-Monte-Carlo

يمثل تقديم نموذج Monte Carlo الوكيلة (AMC) تقدمًا كبيرًا في محاكاة التعلم المعزز لوكلاء الصندوق الأسود، مما يسمح بأخذ عينات مباشرة من السياسات المثلى دون الحاجة إلى تحسين على مستوى المعلمات. تستند هذه الطريقة إلى التكافؤ بين التعلم المعزز والاستدلال البايزي، باستخدام مونت كارلو التسلسلي للتنقل عبر تعقيدات نماذج اللغة الكبيرة (LLMs) في الصندوق الأسود.

La introducción de Agentic Monte Carlo (AMC) representa un avance significativo en la simulación del aprendizaje por refuerzo para agentes de caja negra, permitiendo el muestreo directo de políticas óptimas sin necesidad de optimización a nivel de parámetros. Este método aprovecha la equivalencia entre el aprendizaje por refuerzo y la inferencia bayesiana, utilizando Monte Carlo secuencial para navegar por las complejidades de los modelos de lenguaje de gran tamaño (LLM) en caja negra.

L'introduction de l'Agentic Monte Carlo (AMC) représente une avancée significative dans la simulation de l'apprentissage par renforcement pour les agents en boîte noire, permettant un échantillonnage direct des politiques optimales sans nécessiter d'optimisation au niveau des paramètres. Cette méthode s'appuie sur l'équivalence entre l'apprentissage par renforcement et l'inférence bayésienne, utilisant le Monte Carlo séquentiel pour naviguer dans les complexités des modèles de langage de grande taille (LLM) en boîte noire.

The introduction of Agentic Monte Carlo (AMC) represents a significant advancement in simulating reinforcement learning for black-box agents, allowing for direct sampling from optimal policies without the need for parameter-level optimization. This method leverages the equivalence between reinforcement learning and Bayesian inference, utilizing Sequential Monte Carlo to navigate the complexities of black-box large language models (LLMs).