What is transformer? Meaning and example in Python

What is transformer?

Transformer (also called Transformer architecture): A neural network design that processes a whole sequence at once by letting every position look at every other position through attention, instead of reading step by step like a recurrent network. Stacked identical blocks make it easy to train in parallel on GPUs, which is why it underlies almost all modern language models.

Transformer: an example

GPT, BERT and Llama are all Transformers; a 12-block model reads a 500-token paragraph in one parallel pass rather than 500 sequential steps.

Transformer in Python

A short example you can run (Python 3.11, NumPy). The output below is what it really prints.

import numpy as np
rng = np.random.default_rng(0)

def layer_norm(x):
    return (x - x.mean(-1, keepdims=True)) / (x.std(-1, keepdims=True) + 1e-5)

def attention(x, Wq, Wk, Wv):
    q, k, v = x @ Wq, x @ Wk, x @ Wv
    w = np.exp(q @ k.T / np.sqrt(k.shape[-1])); w /= w.sum(-1, keepdims=True)
    return w @ v

d = 8
x = rng.normal(size=(5, d))                      # 5 tokens, 8 dimensions each
W = [rng.normal(size=(d, d)) * 0.3 for _ in range(3)]
W1, W2 = rng.normal(size=(d, 32)) * 0.3, rng.normal(size=(32, d)) * 0.3
h = x + attention(layer_norm(x), *W)              # attention + residual
out = h + np.maximum(0, layer_norm(h) @ W1) @ W2  # feed-forward + residual
print("in:", x.shape, "out:", out.shape)

Output

in: (5, 8) out: (5, 8)

Related terms

More Transformers LLM architecture terms

Where you learn it

Transformer is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.