MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning

A new method called Mixture of Low-Rank Experts (MoRE) is proposed to enhance multi-task parameter-efficient fine-tuning of large language models by aligning different LoRA ranks with specific tasks and using an adaptive rank selector, leading to improved performance without extra inference costs. Extensive experiments demonstrate that MoRE outperforms traditional LoRA methods across multiple benchmarks, facilitating more efficient multi-task adaptation of LLMs.