arXiv:2605.28774v1 Announce Type: new 
Abstract: Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.

تم تقديم نهج جديد يسمى تحسين سياسة الاستكشاف للوكيل (AXPO) لمعالجة فجوة التفكير-العمل في نماذج الرؤية-اللغة، التي غالبًا ما تواجه صعوبة في استخدام الأدوات أثناء المهام المعقدة. يهدف هذا الأسلوب إلى تحسين فعالية التعلم المعزز من خلال إعادة أخذ عينات من استدعاءات الأدوات واستمرارياتها، مما يعزز إشارة التعلم أثناء التدريب.

Se ha introducido un nuevo enfoque llamado Optimización de Políticas Explorativas de Agentes (AXPO) para abordar la Brecha de Pensamiento-Acción en modelos de visión-lenguaje, que a menudo tienen dificultades con el uso de herramientas durante tareas complejas de razonamiento. Este método busca mejorar la efectividad del aprendizaje por refuerzo al volver a muestrear las llamadas a herramientas y sus continuaciones, mejorando así la señal de aprendizaje durante el entrenamiento.

Une nouvelle approche appelée Agent eXplorative Policy Optimization (AXPO) a été introduite pour traiter le fossé entre la pensée et l'action dans les modèles de vision-langage, qui ont souvent du mal à utiliser des outils lors de tâches de raisonnement complexes. Cette méthode vise à améliorer l'efficacité de l'apprentissage par renforcement en rééchantillonnant les appels d'outils et leurs continuations, améliorant ainsi le signal d'apprentissage pendant l'entraînement.

A new approach called Agent eXplorative Policy Optimization (AXPO) has been introduced to address the Thinking-Acting Gap in vision-language models, which often struggle with tool use during complex reasoning tasks. This method aims to enhance the effectiveness of reinforcement learning by resampling tool calls and their continuations, thereby improving the learning signal during training.

Agent Explorative Policy Optimization for Multimodal Agentic Reasoning

arXiv:2606.01619v2 Announce Type: replace-cross 
Abstract: Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.

تم تقديم إطار عمل جديد يسمى ReSkill لتحسين التعلم المعزز الوكالي (RL) من خلال التوفيق بين إنشاء المهارات وتحسين السياسات. تهدف هذه الطريقة المبتكرة إلى معالجة القيود المفروضة على طرق RL المعززة بالمهارات الموجودة، والتي غالبًا ما تفصل بين تطوير المهارات وتعلم السياسات، مما قد يؤدي إلى استراتيجيات متضاربة. يتضمن ReSkill آليات لمراجعة المهارات ومقارنة خاضعة للرقابة لإصدارات المهارات، مما يعزز قابلية تكيف وكلاء RL.

Se ha introducido un nuevo marco llamado ReSkill para mejorar el aprendizaje por refuerzo agentivo (RL) al reconciliar la creación de habilidades con la optimización de políticas. Este enfoque innovador busca abordar las limitaciones de los métodos de RL aumentados por habilidades existentes, que a menudo separan el desarrollo de habilidades del aprendizaje de políticas, lo que puede llevar a estrategias conflictivas. ReSkill incorpora mecanismos para la revisión de habilidades y la comparación controlada de versiones de habilidades, mejorando así la adaptabilidad de los agentes de RL.

Un nouveau cadre appelé ReSkill a été introduit pour améliorer l'apprentissage par renforcement agentique (RL) en conciliant la création de compétences avec l'optimisation des politiques. Cette approche innovante vise à résoudre les limitations des méthodes RL augmentées par des compétences existantes, qui séparent souvent le développement des compétences de l'apprentissage des politiques, ce qui peut entraîner des stratégies conflictuelles. ReSkill intègre des mécanismes de révision des compétences et de comparaison contrôlée des versions de compétences, améliorant ainsi l'adaptabilité des agents RL.

A new framework called ReSkill has been introduced to enhance agentic reinforcement learning (RL) by reconciling skill creation with policy optimization. This innovative approach aims to address the limitations of existing skill-augmented RL methods, which often separate skill development from policy learning, potentially leading to conflicting strategies. ReSkill incorporates mechanisms for skill revision and controlled comparison of skill versions, enhancing the adaptability of RL agents.

Agent Explorative Policy Optimization for Multimodal Agentic Reasoning

Was this article worth reading? Share it

Ready to build your own newsroom?