0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 17 of 21

Inference

Training happens once; inference runs millions of times in production. Serving LLMs requires optimizing for speed (tokens/second), cost (GPU hours), and quality — different from training optimization.

Training is studying for an exam. Inference is taking the exam — you need to be fast, accurate, and efficient. You can't spend 10 minutes per answer in a timed test.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Inference is the process of running a trained model to generate predictions — for LLMs, this means autoregressive token-by-token generation with careful optimization for latency, throughput, and memory.
  • 2.Autoregressive loop: (1) tokenize input, (2) forward pass through model, (3) get logits for next token, (4) sample/select token (greedy, top-k, top-p, temperature), (5) append to sequence, (6) repeat until EOS or max_tokens.
  • 3.Key metrics: TTFT (time to first token), TPS (tokens per second), latency P50/P99.
  • 4.Optimizations: KV cache (avoid recomputing past keys/values), batching (process multiple requests), quantization (INT8/INT4 weights), speculative decoding (draft model proposes, main model verifies).
  • 5.Frameworks: vLLM, TGI, TensorRT-LLM, llama.cpp.

Real Example

Scenario

Serving GPT-4 API: user sends 500-token prompt. TTFT ~300ms (prefill phase processes all input tokens). Then ~50 tokens/sec generation. Total for 200 output tokens: ~4.3 seconds. Batching 8 requests doubles throughput.

What you would do

In Transformer & ML Foundations, apply Inference to this scenario: Serving GPT-4 API: user sends 500-token prompt. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Inference (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Inference happens in the code.

Inference
1import time  # import dependencies2from transformers import AutoModelForCausalLM, AutoTokenizer  # import dependencies3import torch  # import dependencies4
5model = AutoModelForCausalLM.from_pretrained("gpt2")6tokenizer = AutoTokenizer.from_pretrained("gpt2")7
8prompt = "The future of AI is"9inputs = tokenizer(prompt, return_tensors="pt")10
11start = time.time()12with torch.no_grad():13    outputs = model.generate(14        **inputs,15        max_new_tokens=50,16        do_sample=True,17        temperature=0.7,18        top_p=0.9,19    )20elapsed = time.time() - start21tokens = outputs.shape[1] - inputs.input_ids.shape[1]22print(f"Generated {tokens} tokens in {elapsed:.2f}s ({tokens/elapsed:.1f} tok/s)")  # show output for debugging

Cheat Sheet

Quick recap

quick ref
  • •Prefill: process prompt (TTFT)
  • •Decode: generate tokens (TPS)
  • •Greedy: argmax, deterministic
  • •top_p=0.9: nucleus sampling
  • •temperature: 0=factual, 1=creative
  • •torch.no_grad() for inference

Common Mistakes

  • ✕Not using torch.no_grad() during inference — wastes memory
  • ✕Recomputing full attention without KV cache — O(n²) per token
  • ✕Using high temperature for factual extraction tasks
  • ✕Ignoring batch size effects on latency vs throughput