Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking

A new method called Decom-Renorm-Merge (DRM) has been proposed to improve neural network model merging by using Singular Value Decomposition to align weight matrices into a joint space, enabling more effective merging across various architectures. DRM outperforms existing techniques in both full finetuning and low-rank adaptation scenarios, with renormalization identified as a key factor for creating a robust merging process.