Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

This paper reviews and benchmarks memory-efficient pre-training methods for large language models, highlighting that full-rank training generally yields the best performance but that high-rank updates in low-rank approaches can improve results. It also introduces two techniquesweight refactorization and momentum resetthat enhance low-rank methods, enabling comparable or better performance with significantly reduced memory usage.