0
0Neural Networks1Gradient Descent2Backpropagation3Activation Functions4Loss Functions5Optimizers6Attention7Self Attention8Multi Head Attention9Encoder10Decoder11Positional Encoding12Transformer Architecture13Word2Vec14BERT15GPT16Inference17KV Cache18RoPE19MoE20Quantization
Transformer & ML Foundations
Phase 1.1 · OptionalOptionalModule 21 of 21

Quantization

A 70B parameter model in FP16 needs ~140GB GPU memory — impossible on consumer hardware. INT4 quantization reduces this to ~35GB, making large models runnable on a single high-end GPU or even CPU.

Storing a photo as PNG (lossless, large) vs JPEG (compressed, smaller, nearly identical to the eye). Quantization is JPEG for neural network weights — smaller file, slightly less precision, usually fine.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%
Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

  • 1.Quantization reduces model weight precision from FP16/FP32 to INT8 or INT4 — shrinking model size and accelerating inference with minimal quality loss, enabling local LLM deployment. Post-Training Quantization (PTQ): quantize after training — GPTQ, AWQ, GGUF formats.
  • 2.Quantization-Aware Training (QAT): simulate quantization during training for better accuracy. INT8: 2× compression vs FP16.
  • 3.INT4: 4× compression. Methods: GPTQ (layer-wise optimal quantization), AWQ (protect salient weights), GGUF (llama.cpp format with k-quant variants: Q4_K_M, Q5_K_S).
  • 4.Tradeoffs: lower bits = smaller/faster but more quality degradation. Q4_K_M is popular sweet spot for local models.
  • 5.Calibration dataset needed for PTQ to minimize error.

Real Example

Scenario

Llama-3-70B: FP16 = 140GB (needs 2× A100). Q4_K_M GGUF = ~40GB (fits 1× A100 or Mac M2 Ultra). Quality: MMLU 79.5 (FP16) vs 78.1 (Q4) — 1.4 point drop for 3.5× size reduction.

What you would do

In Transformer & ML Foundations, apply Quantization to this scenario: Llama-3-70B: FP16 = 140GB (needs 2× A100). Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Quantization (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Quantization happens in the code.

Quantization
1# Using Ollama with quantized models2# ollama pull llama3:70b-instruct-q4_K_M3
4# HuggingFace GPTQ quantization5from transformers import AutoModelForCausalLM, GPTQConfig  # import dependencies6
7gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)8model = AutoModelForCausalLM.from_pretrained(9    "meta-llama/Llama-3-8B",10    quantization_config=gptq_config,  # key line for Quantization11    device_map="auto",12)13
14# llama.cpp conversion15# python convert.py models/llama-3-8b --outtype q4_k_m

Cheat Sheet

Quick recap

quick ref
  • •INT4 = 4× smaller than FP16
  • •GPTQ: layer-wise optimal PTQ
  • •AWQ: protect salient weights
  • •GGUF: llama.cpp format
  • •Q4_K_M: recommended default
  • •ollama pull model:q4_K_M

Common Mistakes

  • ✕Using INT2 or aggressive quantization without quality testing
  • ✕Skipping calibration dataset for PTQ — poor accuracy
  • ✕Comparing quantized model quality to FP16 without same eval setup
  • ✕Assuming quantization speeds up training — it only helps inference