RA3: Mid-Training with Temporal Action Abstractions for Faster Reinforcement Learning (RL) Post-Training in Code LLMs

Apple's recent research introduces RA3 (Reasoning as Action Abstractions), an Expectation-Maximization (EM)-style procedure designed to enhance reinforcement learning (RL) in code large language models (LLMs). RA3 learns temporally consistent latent actions from expert traces during mid-training, enabling the pruning of the action space to a compact, near-optimal subset and reducing the effective planning horizon, which accelerates RL convergence and improves downstream performance on benchmarks like HumanEval and MBPP by approximately 8 and 4 points, respectively. This approach formalizes the role of mid-training in shaping