Weight Spectra Induced Efficient Model Adaptation

This research investigates how fine-tuning large models alters their weight matrices, revealing that it mainly amplifies top singular values and reorients dominant singular vectors, indicating task-specific knowledge is embedded in low-dimensional subspaces. Building on these findings, the authors propose a method that modulates top singular directions through learnable rescaling, improving performance across tasks while maintaining the model's global structure.