Two Is Better Than One: Rotations Scale LoRAs

A new gating method called RadarGate enhances the scalability and generalization of LoRA-based Mixture-of-Experts in large language models by introducing rotational operations among LoRA representations, enabling richer feature interactions. Extensive experiments across multiple benchmarks demonstrate its effectiveness in addressing underfitting and poor generalization issues as the number of LoRAs increases.