We hypothesize that the update matrices in language model adaptationalso have a low “intrinsic rank”, leading to our proposedLow-RankAdaptation (LoRA) approach.LoRA allows us to train every dense layer in a neural network indirectly by injecting and optimizingrank decomposition matrices of the dense layer’s update instead, while keeping the orig.