Accelerating RLHF Training with Reward Variance Increase

A new method called reward adjustment model is proposed to enhance the efficiency of reinforcement learning from human feedback (RLHF) in training large language models, by increasing reward variance while preserving preferences. The approach integrates into the group relative policy optimization (GRPO) algorithm, resulting in the more efficient GRPOVI, which significantly accelerates RLHF training compared to existing methods.