LLM Interview Questions (2026)

Get the basics wrong and the interview is over in 10 minutes.

• →Q: Explain self-attention. A: Q·Kᵀ / √d_k → softmax → V. Mention scaling, multi-head, why it parallelizes.
• →Q: Decoder-only vs encoder-decoder? A: Decoder-only = causal mask, autoregressive (GPT/Claude). Encoder-decoder = separate encoder + cross-attention (T5).
• →Q: How does tokenization affect a model? A: BPE/WordPiece; vocab size trade-off; why 'strawberry' has 3 r's the model can't count; tokens, not characters, are the unit of cost.
• →Q: What is a context window and why is it expensive? A: Max tokens the attention can process; cost grows quadratically without FlashAttention; KV cache memory grows linearly.
• →Q: Top-k vs top-p vs temperature? A: Three sampling controls — k caps candidate count, p caps cumulative prob, temperature flattens or sharpens the distribution.

RAG is on every AI-engineer interview now.

• →Q: How would you build RAG from scratch? A: Chunk → embed → store in vector DB → retrieve top-k → rerank → generate. Mention chunking strategy, hybrid retrieval, eval.
• →Q: How do you evaluate RAG quality? A: Faithfulness + relevance + answer correctness; RAGAs framework; LLM-as-judge; a golden set of (q, ideal-passage, ideal-answer).
• →Q: When does RAG fail? A: Multi-hop queries, ambiguous queries, when chunk size mismatches query granularity, when reranker is missing.
• →Q: What's the difference between CRAG, GraphRAG, Adaptive RAG? A: CRAG = critic checks retrieval. GraphRAG = traverse a knowledge graph. Adaptive = router decides whether/how to retrieve.
• →Q: RAG vs long-context — which wins? A: RAG is cheaper and more grounded; long context is simpler to build. Hybrid (RAG + 200k context) is the production answer.

• →Q: When fine-tune vs prompt vs RAG? A: Fine-tune for behavior (tone, format), RAG for facts, prompts for everything else. Often combine.
• →Q: Explain LoRA. A: Add rank-r matrices B·A to each frozen weight; train only B and A; 10,000× fewer params; same quality.
• →Q: What is RLHF? A: Train a reward model from preference pairs; use PPO to update the policy LLM to maximize reward; risks reward hacking.
• →Q: DPO vs PPO for alignment? A: DPO skips the reward model and trains directly on preferences; simpler and often equal quality.
• →Q: How would you detect reward hacking? A: Watch for unexpected behavior changes, off-policy rollouts, KL divergence from base model, qualitative regression tests.

• →Q: What is the KV cache? A: Cached K and V tensors per generated token so subsequent generation skips re-computing them; the reason inference is bandwidth-bound.
• →Q: How does PagedAttention work? A: vLLM's KV-cache management; virtual-memory-style paging eliminates fragmentation; key reason vLLM is fast.
• →Q: What is speculative decoding? A: Small draft model proposes N tokens; big model verifies in parallel; 2-3× speedup with zero quality loss.
• →Q: How do you quantize an LLM? A: PTQ (GPTQ, AWQ) for inference, QAT for training; trade ~1-2% accuracy for 2-4× memory savings.
• →Q: How do you serve thousands of LoRA fine-tunes from one base? A: S-LoRA / Punica / Lorax — hot-swap adapters per request, share the base model.

• →Q: Design a customer-support chatbot. A: RAG over docs, conversational memory, function-calling for ticket lookup, guardrails, eval loop with feedback.
• →Q: Design a multi-tenant LLM API. A: Per-tenant prompts/fine-tunes, rate limits, billing, isolation via LoRA hot-swap, noisy-neighbor controls.
• →Q: How would you reduce LLM cost by 50%? A: Caching, prompt compression, model routing (small for easy, large for hard), batching, quantization.
• →Q: How would you detect drift in a production LLM app? A: PSI / KL on input embeddings, performance metrics on golden set, sample-based human eval, regression alerts.
• →Q: How would you handle prompt injection? A: Pre-LLM filters, output structure validation, never trust retrieved content as instructions, principle of least privilege for tools.