LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration

The introduction of Lexicographically Projected Policy Gradient Reinforcement Learning (LPPG-RL) marks a significant advancement in the field of multi-objective reinforcement learning (MORL). Traditional MORL methods often rely on heuristic threshold tuning or are limited to discrete domains, which can hinder their effectiveness in real-world applications. LPPG-RL overcomes these limitations by employing sequential gradient projections, reformulating the projection step as an optimization problem, and ensuring compatibility with all policy gradient algorithms in continuous spaces. This innovative approach not only accelerates convergence and enhances stability but also provides theoretical guarantees for policy improvement. The proposed method is expected to deliver substantial speedups, particularly for small- to medium-scale instances, making it a promising tool for tackling complex multi-objective problems across various domains.