AI engineering glossary · Transformers LLM architecture
What is llama?
Llama: A family of open-weight Transformer language models released by Meta. They are decoder-only models that use pre-norm with RMSNorm, SwiGLU MLPs and rotary position embeddings, and have become a common base for fine-tuning and for other open models.
Llama: an example
A developer fine-tunes a Llama model on Hindi customer-support chats using LoRA.
Related terms
More Transformers LLM architecture terms
- LM head
- Load-balancing loss
- Log probabilities
- Logit lens
- Long-context model
- Longformer
- Mamba
- Maximum output tokens
Where you learn it
Llama is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.