What is loRA? Meaning and example in Python

What is loRA?

LoRA (also called Low-Rank Adaptation): Low-Rank Adaptation, a fine-tuning technique that freezes the original weight matrix W and learns a small update written as the product of two thin matrices B and A, so the effective weight is W + (alpha/r) * B*A. It trains far fewer parameters and the update can be merged into W afterwards.

LoRA: an example

For a 4096x4096 matrix, rank 8 needs 2 x 4096 x 8 = 65,536 trainable values instead of about 16.8 million.

LoRA in Python

A short example you can run (Python 3.11, NumPy). The output below is what it really prints.

import numpy as np
rng = np.random.default_rng(0)

d, r = 1024, 8
W = rng.normal(size=(d, d))            # frozen pretrained weight
A = rng.normal(size=(r, d)) * 0.01     # trainable, rank r
B = np.zeros((d, r))                   # trainable, starts at zero
W_adapted = W + B @ A                  # the update has rank <= r

print("full fine-tune params:", W.size)
print("LoRA params:", A.size + B.size, f"({(A.size + B.size) / W.size:.2%})")

Output

full fine-tune params: 1048576
LoRA params: 16384 (1.56%)

Related terms

More LLM training alignment terms

Where you learn it

LoRA is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.