Calibrated Value-Aware Model Learning with Stochastic Environment Models

This paper examines the limitations of the MuZero loss and similar value-aware model learning methods, revealing that they are uncalibrated surrogate losses that may not accurately recover the true model and value functions. The authors propose corrective measures and analyze the impact of model architectures and auxiliary losses, finding that while deterministic models can suffice for value prediction, calibrated stochastic models offer advantages.