Scalable Complexity Control Facilitates Reasoning Ability of LLMs

Adjusting model complexity through initialization rate and weight decay enhances the scaling performance of large language models across different sizes and data amounts. Using a constant initialization rate, rather than fixed std, accelerates improvements, suggesting complexity control as a key avenue for advancing LLM capabilities.