RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs

The recent release of DeepSeek R1 has sparked interest in reinforcement learning (RL) post-training for large language models (LLMs). However, a critical analysis reveals that the structural assumptions underlying these methods, particularly the modeling of LLM training as a Markov Decision Process (MDP), result in a degenerate MDP. This leads to the conclusion that the RL approach is effectively equivalent to outcome-driven supervised learning. Experiments conducted on benchmarks such as GSM8K and Countdown demonstrate that iterative supervised fine-tuning can achieve performance comparable to that of GRPO-based training. This finding is crucial as it challenges the prevailing narrative that RL significantly enhances reasoning abilities in LLMs, suggesting that the improvements attributed to RL may not be as substantial as previously thought.