"Normalized Low-Rank Adaptation" by Jiale Kang , Ziyin Yue , Zheng Zhan , Yangyi Huang , Weiyang Liu
TLDR:
The text describes a technique called Normalized Low-Rank Adaptation (NoRA) that stabilizes LoRA training by normalizing down-projection matrices. NoRA accelerates convergence, enhances performance, and improves training stability across different learning tasks like pretraining, supervised finetuning, and reinforcement learning. This method does not require extra parameters or inference cost, making it a simple and effective strategy to optimize the training dynamics of LoRA.
Read Paper / Blog
huggingface.co
Paper page - Normalized Low-Rank Adaptation
Join the discussion on this paper page
1September 2, 2026 40 1
