Vision Models
Much of the world's information is visual — charts, screenshots, handwritten notes, product photos. Vision-capable LLMs bridge the text-only gap, enabling multimodal understanding and analysis.
Before vision models, LLMs were like a person who could only read descriptions of photos. Vision models give them eyes — they can look at the actual image and describe, analyze, and reason about what they see.
Visual Workflows
Start here — scroll inside each diagram frame to explore, then use + / − to zoom up to 200% if needed.
Overview
Scroll inside the frame to explore · use + / − to zoom up to 200%
Scroll inside the frame to explore · use + / − to zoom up to 200%
Image + Text in One Call
Scroll inside the frame to explore · use + / − to zoom up to 200%
Resize first — extra pixels cost tokens and rarely add accuracy.
Key Takeaways
- 1.Vision models enable LLMs to understand images — analyzing photos, diagrams, screenshots, and documents visually, powering applications from receipt scanning to UI testing. Approach: GPT-4o/Gemini/Claude accept images directly in messages (native multimodal).
- 2.Alternative: separate vision encoder (CLIP) → projection layer → LLM (LLaVA architecture). Use cases: image captioning, OCR/document analysis, chart interpretation, UI screenshot analysis, visual QA.
- 3.Image input: URL, base64, or file upload. Resolution matters: higher res = better detail but more tokens.
- 4.GPT-4o processes images at multiple scales. Tips: ask specific questions, reference regions, combine with text context.
Real Example
Scenario
Receipt scanning: photograph receipt → send to GPT-4o with 'Extract vendor, date, items, and total as JSON' → model reads the image, returns structured data. No separate OCR pipeline needed.
What you would do
In LLM Engineering & APIs, apply Vision Models to this scenario: Receipt scanning: photograph receipt → send to GPT-4o with 'Extract vendor, date, items, and total as JSON' → model reads the image, returns structured data. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.
Practice Task
Open the Code Walkthrough below and run it locally. Change one parameter related to Vision Models (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.
Code Walkthrough
Highlighted lines show where Vision Models happens in the code.
1from openai import OpenAI # import dependencies2import base64 # import dependencies3
4client = OpenAI() # create API client5
6def encode_image(path): # define a reusable function7 with open(path, "rb") as f:8 return base64.b64encode(f.read()).decode() # return the result9
10response = client.chat.completions.create( # call the API11 model="gpt-4o",12 messages=[{13 "role": "user",14 "content": [15 {"type": "text", "text": "Describe this image and extract any text you see."},16 {"type": "image_url", "image_url": {17 "url": f"data:image/png;base64,{encode_image('screenshot.png')}"18 }},19 ],20 }],21 max_tokens=500,22)23print(response.choices[0].message.content) # show output for debuggingCommands
Commands to Remember
content=[{type:'text', text: question}, {type:'image_url', image_url:{url: data_url}}] # GPT-4o visionbase64.b64encode(open('photo.png','rb').read()).decode() # inline imageAsk a specific question, not just 'what is this?' # better answersResize large screenshots before sending # tokens and latency
Cheat Sheet
Quick recap
quick ref- •type: image_url in content
- •base64 encode local images
- •model: gpt-4o for vision
- •High res = more tokens = more cost
- •Ask specific questions about images
- •LLaVA for open-source vision
Common Mistakes
- ✕Sending unnecessarily high-resolution images — wasting tokens
- ✕Vague questions about images — 'what is this?' vs specific asks
- ✕Assuming perfect OCR — always validate extracted data
- ✕Not handling image format/size API limits