AI engineering glossary · Transformers LLM architecture
What is RWKV?
RWKV: A language model architecture that can be trained in parallel like a Transformer but run like a recurrent network at inference, with a fixed-size state instead of a growing cache. It aims to keep quality while making long generation cheap.
RWKV: an example
An RWKV model running on a small device keeps one fixed-size state no matter how long the conversation grows.
Related terms
More Transformers LLM architecture terms
- Scaled dot-product attention
- Self-attention
- Sequence length
- Sequence-to-sequence model
- Shared expert
- Sinusoidal positional encoding
- Sliding window attention
- Sparse attention
Where you learn it
RWKV is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.