AI engineering glossary · Transformers LLM architecture

What is self-attention?

Self-attention (also called Intra-attention): Attention where the queries, keys and values all come from the same sequence, so each token gathers information from the other tokens around it. The result is a new representation for each token that reflects its context, such as which noun a pronoun refers to.

Level: beginnerTopic: Transformers LLM architecture

Self-attention: an example

In 'The bank raised its rates', self-attention lets 'its' pull information from 'bank', helping the model treat 'bank' as a financial institution.

Self-attention in Python

A short example you can run (Python 3.11, NumPy). The output below is what it really prints.

import numpy as np

x = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])   # 3 tokens
scores = x @ x.T / np.sqrt(x.shape[1])
weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)
print(np.round(weights, 3))
print(np.round(weights @ x, 3))                      # each token mixes in the others

Output

[[0.401 0.198 0.401]
 [0.198 0.401 0.401]
 [0.248 0.248 0.503]]
[[0.802 0.599]
 [0.599 0.802]
 [0.752 0.752]]

Related terms

More Transformers LLM architecture terms

Where you learn it

Self-attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.