NVIDIA Researchers Propose Reinforcement Learning Pretraining (RLP): Reinforcement as a Pretraining Objective for Building Reasoning During Pretraining

NVIDIA AI has developed Reinforcement Learning Pretraining (RLP), a novel approach that integrates reinforcement learning directly into the pretraining phase of language models, rather than applying it post-training. This method treats short chain-of-thought (CoT) sequences as actions sampled before next-token prediction, rewarding them based on the information gain they provide, measured against an EMA-based no-think baseline. The approach employs a single shared-parameter network to sample CoT policies and score subsequent tokens, with a slowly updated EMA teacher network providing a counterfactual baseline, enabling dense, position-wise rewards without the