Phase 1.1 · OptionalOptionalModule 16 of 21

GPT

A single generative model that learns from unlabeled text can perform any language task via prompting — no task-specific architecture needed. Scaling GPT from 117M (GPT-1) to 175B (GPT-3) parameters unlocked emergent capabilities like in-context learning and chain-of-thought reasoning.

A student who learned by reading billions of books and can now write essays, answer questions, and solve problems — all by continuing text naturally, without being explicitly taught each skill.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.GPT (Generative Pre-trained Transformer) is a decoder-only autoregressive model trained to predict the next token — the architecture family behind ChatGPT and virtually all modern text-generating LLMs. Architecture: decoder-only transformer.
2.Pre-training: causal language modeling — predict token t given tokens 1..t-1. Training objective: minimize cross-entropy over next-token prediction on massive text corpora.
3.Post-training: SFT (supervised fine-tuning on instruction-response pairs) → RLHF (reinforcement learning from human feedback) for alignment. Key insight from GPT-3: in-context learning — model learns from examples in the prompt without weight updates.
4.Scaling: GPT-1 (117M) → GPT-2 (1.5B) → GPT-3 (175B) → GPT-4 (MoE, undisclosed). Each scale jump unlocks new capabilities.

Real Example

Scenario

GPT-3 in-context learning: prompt 'Translate to French: hello → bonjour, goodbye →' → model generates 'au revoir' without any fine-tuning, learning the pattern from the single example in context.

What you would do

In Transformer & ML Foundations, apply GPT to this scenario: GPT-3 in-context learning: prompt 'Translate to French: hello → bonjour, goodbye →' → model generates 'au revoir' without any fine-tuning, learning the pattern from the single example in context. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to GPT (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where GPT happens in the code.

GPT

1from openai import OpenAI  # import dependencies2
3client = OpenAI()  # create API client4
5# GPT-4 via API — decoder-only generation6response = client.chat.completions.create(  # call the API7    model="gpt-4o",8    messages=[9        {"role": "system", "content": "You are a helpful assistant."},10        {"role": "user", "content": "Explain GPT in one sentence."},11    ],12    temperature=0.7,13    max_tokens=100,14)15print(response.choices[0].message.content)  # show output for debugging16
17# In-context learning example18few_shot = """Classify sentiment:19Text: I love this! Sentiment: positive20Text: This is terrible. Sentiment: negative21Text: Best day ever! Sentiment:"""

Cheat Sheet

Quick recap

quick ref

•GPT: decoder-only, causal LM
•Pre-train → SFT → RLHF
•In-context learning: examples in prompt
•GPT-3: 175B params
•temperature: 0=deterministic, 1=creative
•max_tokens limits output length

Common Mistakes

✕Expecting small GPT models to do few-shot learning — needs scale
✕Confusing pre-training with fine-tuning objectives
✕Using high temperature for factual/reasoning tasks
✕Assuming GPT 'understands' — it predicts likely next tokens