Quasi-Newton Compatible Actor-Critic for Deterministic Policies

arXiv — cs.LG•Thursday, November 13, 2025 at 5:00:00 AM

The recent paper titled 'Quasi-Newton Compatible Actor-Critic for Deterministic Policies' presents a novel second-order deterministic actor-critic framework that builds upon the classical deterministic policy gradient method. By integrating curvature information from the performance function, the authors introduce a quadratic critic that not only preserves the true policy gradient but also approximates the performance Hessian. This innovation allows for a least-squares temporal difference learning scheme to efficiently estimate the quadratic critic parameters. Consequently, the framework enables a quasi-Newton actor update, resulting in faster convergence compared to traditional first-order methods. The numerical examples provided in the paper demonstrate significant improvements in convergence and performance over standard deterministic actor-critic baselines. This general applicability to any differentiable policy class positions the proposed method as a substantial advancement in th…

— via World Pulse Now AI Editorial System

Read Original

Was this article worth reading? Share it

Recommended Readings

arXiv — cs.LG5 hours ago

EvoLM: In Search of Lost Language Model Training Dynamics

PositiveArtificial Intelligence

EvoLM is a new model suite designed to analyze the training dynamics of language models (LMs) across various stages, including pre-training and fine-tuning. By training over 100 LMs with 1B and 4B parameters, EvoLM provides insights into the effectiveness of design choices and their impact on both language modeling and problem-solving capabilities. Key findings emphasize the diminishing returns of excessive pre-training and the importance of continued pre-training to mitigate forgetting during domain-specific tasks.

Read full article

via arXiv — cs.LG

arXiv — cs.LG5 hours ago

Knowledge vs. Experience: Asymptotic Limits of Impatience in Edge Tenants

NeutralArtificial Intelligence

The study investigates the impact of two information feeds, a closed-form Markov estimator and an online trained actor-critic, on reneging and jockeying behaviors in a dual M/M/1 system. It reveals that with unequal service rates and total-time patience, total wait increases linearly, leading to inevitable abandonment. The probability of successful jockeying diminishes as backlog increases. Both information models converge to the same asymptotic limits under certain conditions, highlighting the importance of value-of-information in finite regimes.

Read full article

via arXiv — cs.LG

arXiv — cs.CL2 days ago

LDC: Learning to Generate Research Idea with Dynamic Control

PositiveArtificial Intelligence

Recent advancements in large language models (LLMs) highlight their potential in automating scientific research ideation. Current methods often produce ideas that do not meet expert standards of novelty, feasibility, and effectiveness. To address these issues, a new framework is proposed that combines Supervised Fine-Tuning (SFT) and controllable Reinforcement Learning (RL) to enhance the quality of generated research ideas through a two-stage approach.

Read full article

via arXiv — cs.CL

arXiv — cs.LG2 days ago

Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning

PositiveArtificial Intelligence

The paper titled 'Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning' addresses the challenges of high-variance return estimates in reinforcement learning algorithms. It highlights that well-designed behavior policies can collect off-policy data, leading to lower variance return estimates. This finding suggests that on-policy data collection is not optimal for variance, and the authors extend this insight to online reinforcement learning, where policy evaluation and improvement occur simultaneously.

Read full article

via arXiv — cs.LG

arXiv — cs.CL2 days ago

Thinker: Training LLMs in Hierarchical Thinking for Deep Search via Multi-Turn Interaction

PositiveArtificial Intelligence

The article presents Thinker, a hierarchical thinking model designed to enhance the reasoning capabilities of large language models (LLMs) through multi-turn interactions. Unlike previous methods that relied on end-to-end reinforcement learning without supervision, Thinker allows for a more structured reasoning process by breaking down complex problems into manageable sub-problems. Each sub-problem is represented in both natural language and logical functions, improving the coherence and rigor of the reasoning process.

Read full article

via arXiv — cs.CL

arXiv — cs.CL2 days ago

From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Models

NeutralArtificial Intelligence

Recent advancements in large language models (LLMs) have shifted the focus of reasoning as a benchmark for intelligence evaluation. This article critiques the uniform reasoning strategies employed by current LLMs, which often generate lengthy reasoning for simple tasks while struggling with complex ones. It introduces the concept of adaptive reasoning, which allows models to adjust their reasoning efforts based on task difficulty and uncertainty, and outlines three key contributions to understanding this approach.

Read full article

via arXiv — cs.CL

arXiv — cs.LG2 days ago

DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control

PositiveArtificial Intelligence

The paper titled 'DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control' discusses the introduction of a disturbance-augmented Markov decision process (DAMDP) to enhance reinforcement learning in robotic control. It addresses the challenges of sim2real transfer, where models trained in simulation often fail to perform effectively in real-world scenarios due to discrepancies in system dynamics. The proposed approach aims to improve the robustness and stabilization of control responses in robotic systems.

Read full article

via arXiv — cs.LG

arXiv — cs.LG2 days ago

Mining--Gym: A Configurable RL Benchmarking Environment for Truck Dispatch Scheduling

PositiveArtificial Intelligence

Mining-Gym is introduced as a configurable, open-source benchmarking environment aimed at optimizing truck dispatch scheduling in mining operations. The dynamic and stochastic nature of mining environments, characterized by uncertainties such as equipment failures and variable haul cycle times, poses challenges to traditional optimization methods. By leveraging Reinforcement Learning (RL), Mining-Gym provides a platform for training, testing, and evaluating RL algorithms, enhancing the efficiency and adaptability of decision-making in mining logistics.

Read full article

via arXiv — cs.LG