arXiv:2511.17598v2 Announce Type: replace-cross 
Abstract: Algorithms developed under stationary Markov Decision Processes (MDPs) often face challenges in non-stationary environments, and infinite-horizon formulations may not directly apply to finite-horizon tasks. To address these limitations, we introduce the Non-stationary and Varying-discounting MDP (NVMDP) framework, which naturally accommodates non-stationarity and allows discount rates to vary with time and transitions. Infinite-horizon, stationary MDPs emerge as special cases of NVMDPs for identifying an optimal policy, and finite-horizon MDPs are also subsumed within the NVMDP formulations. Moreover, NVMDPs provide a flexible mechanism to shape optimal policies, without altering the state space, action space, or the reward structure. We establish the theoretical foundations of NVMDPs, including assumptions, state- and action-value formulation and recursion, matrix representation, optimality conditions, and policy improvement under finite state and action spaces. Building on these results, we adapt dynamic programming and generalized Q-learning algorithms to NVMDPs, along with formal convergence proofs. For problems requiring function approximation, we extend the Policy Gradient Theorem and the policy improvement bound in Trust Region Policy Optimization (TRPO), offering proofs in both scalar and matrix forms. Empirical evaluations in a non-stationary gridworld environment demonstrate that NVMDP-based algorithms successfully recover optimal trajectories under multiple reward and discounting schemes, whereas original Q-learning fails. These results collectively show that NVMDPs provide a theoretically sound and practically effective framework for reinforcement learning, requiring only minor algorithmic modifications while enabling robust handling of non-stationarity and explicit optimal policy shaping.

يقدم إطار عمليات اتخاذ القرار ماركوف غير الثابتة والمتغيرة الخصم (NVMDP) حلاً للتحديات التي تواجهها العمليات الثابتة في البيئات غير الثابتة، مما يسمح بتغيير معدلات الخصم مع مرور الوقت والانتقالات. يشمل هذا الإطار المهام ذات الأفق غير المحدود والمحدود، مما يوفر نهجًا أكثر مرونة للتعلم المعزز.

La introducción del marco de Procesos de Decisión de Markov No Estacionarios y de Descuento Variable (NVMDP) aborda las limitaciones que enfrentan los Procesos de Decisión de Markov estacionarios tradicionales en entornos no estacionarios, permitiendo que las tasas de descuento varíen con el tiempo y las transiciones. Este marco abarca tanto tareas de horizonte infinito como de horizonte finito, proporcionando un enfoque más adaptable para el aprendizaje por refuerzo.

L'introduction du cadre des Processus de Décision de Markov Non-stationnaires et à Taux d'Actualisation Variable (NVMDP) répond aux limitations rencontrées par les Processus de Décision de Markov stationnaires traditionnels dans des environnements non-stationnaires, permettant des taux d'actualisation variables au fil du temps et des transitions. Ce cadre englobe à la fois les tâches à horizon infini et à horizon fini, offrant une approche plus adaptable à l'apprentissage par renforcement.

The introduction of the Non-stationary and Varying-discounting Markov Decision Processes (NVMDP) framework addresses the limitations faced by traditional stationary Markov Decision Processes (MDPs) in non-stationary environments, allowing for varying discount rates over time and transitions. This framework encompasses both infinite-horizon and finite-horizon tasks, providing a more adaptable approach to reinforcement learning.

Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learning

Was this article worth reading? Share it

LucidQuery AI

DeployMCP

Synthx

Deltabadger

Hypertune

Mirava

Ready to build your own newsroom?