Off-Policy Reinforcement Learning RL with KL Divergence Yields Superior Reasoning in Large Language Models

Reinforcement learning, particularly with policy gradient methods like PPO, has improved LLM reasoning, with KL regularization playing a key role in stabilizing training by limiting policy shifts. Advances include exploring different KL variants and training strategies such as reward model optimization and direct preference methods like DPO, which enhance alignment and reasoning capabilities while reducing computational costs.