LLM interview questions and answers for AI engineers

These are questions interviewers ask junior and mid-level AI engineers about large language models. The answers are short on purpose: give the core idea first, then add an example from something you built.

Fundamentals

1. What is a large language model? A neural network, usually a transformer, trained on very large amounts of text to predict the next token. That simple objective, at scale, gives it the ability to follow instructions, summarise, translate and write code.

2. What is a token? The unit a model reads and writes: often a word piece, not a whole word. Prices, context limits and speed are all measured in tokens.

3. What is the context window? The maximum number of tokens the model can consider at once, covering both the input and the output. Anything outside it is invisible to the model.

4. How does attention work, in one sentence? For each token, the model computes how much every other token in the context matters to it and builds a weighted mix of their information.

5. What do temperature and top-p control? Randomness in choosing the next token. Low temperature makes output more predictable; higher values make it more varied. Top-p limits choices to the most likely tokens whose probabilities add up to p.

6. What is a hallucination? A fluent answer that is not supported by facts or by the given sources. Retrieval, citations, constrained output and evaluation reduce it; nothing removes it completely.

Prompting and outputs

7. Zero-shot vs few-shot prompting? Zero-shot gives only instructions; few-shot adds a handful of worked examples in the prompt to show the format or style.

8. How do you get reliable JSON from an LLM? Use the provider's structured-output or tool-calling feature with a schema, validate the result in code, and retry or repair when validation fails.

9. What is a system prompt? Instructions placed before the conversation that set the model's role, rules and output format.

10. What is prompt injection? Text in the input (for example a web page or an email) that tries to override your instructions. Defences include separating instructions from data, limiting tool permissions and checking outputs before acting.

Embeddings and RAG

11. What is an embedding? A list of numbers representing the meaning of a text, so similar meanings sit close together and can be compared, often with cosine similarity.

12. What is RAG? Retrieval-augmented generation: search a knowledge source for relevant passages, then give them to the model with the question so the answer is grounded in them.

13. What is chunking and why does size matter? Splitting documents into pieces before embedding. Chunks too small lose context; too large dilute relevance and waste the context window.

14. Vector search vs keyword search: when do you use hybrid search? Vector search finds meaning; keyword search (such as BM25) finds exact terms like product codes. Hybrid search combines both and is a strong default for business documents.

15. What is reranking? A second, more accurate model that reorders the top retrieved passages before they go to the LLM.

16. How do you evaluate a RAG system? Separately: did retrieval find the right passages (recall, precision at k), and is the answer faithful to them and relevant to the question? Use a labelled test set, not a few manual checks.

Fine-tuning

17. When would you fine-tune instead of prompting or RAG? Fine-tune to change behaviour or format consistently, or to make a smaller model good at one narrow task. Use RAG to give the model knowledge that changes.

18. What is LoRA? Low-rank adaptation: train small extra matrices instead of all the model's weights, which makes fine-tuning cheaper and lets you swap adapters.

19. What is RLHF, briefly? Reinforcement learning from human feedback: people rank model outputs, a reward model learns those preferences, and the LLM is tuned to score well on it.

20. What is quantization? Storing weights with fewer bits (for example 4-bit) to cut memory and speed up inference, with a small loss in quality.

Agents

21. What is an AI agent? A system where the model decides the next step, calls tools such as search or code execution, reads the results and repeats until it reaches a goal.

22. What is function or tool calling? The model returns a structured request to call a function with arguments; your code runs it and sends back the result.

23. What is the Model Context Protocol (MCP)? An open standard for connecting AI applications to tools and data sources through one common interface.

24. How do you make an agent safe? Least-privilege tools, limits on steps and spending, human approval for risky actions, logging, and tests for agent behaviour.

Evaluation and production

25. How do you evaluate an LLM application? Build a test set of real inputs with expected properties, score outputs automatically where possible (exact checks, schema checks, LLM-as-a-judge with a rubric), and run it on every change.

26. What is LLM-as-a-judge and its risk? Using a model to grade outputs against a rubric. It scales well, but it can be biased, so check it against human labels.

27. How do you reduce cost and latency? Smaller models where they suffice, caching, shorter prompts, batching, streaming responses and limiting output length.

28. What do you monitor in production? Latency, errors, cost per request, output quality on sampled traffic, safety flags and drift in the kind of inputs users send.

29. What is a KV cache? Stored attention keys and values from earlier tokens, so the model does not recompute them for each new token. It speeds up generation but uses memory.

30. Walk me through a project you built. Have one ready: the problem, your design choices, how you measured quality, what failed and what you changed. This answer matters more than any definition.

Every term above has a short definition in our AI engineering glossary.