Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics

This study examines the effects of non-zero initialization in Low-Rank Adaptation (LoRA) fine-tuning, challenging the traditional practice of starting with zero matrices. Results show that initializing both matrices to non-zero values enhances robustness to learning rate issues without compromising performance, supported by theoretical analysis and extensive experiments.