Vision Models

Much of the world's information is visual — charts, screenshots, handwritten notes, product photos. Vision-capable LLMs bridge the text-only gap, enabling multimodal understanding and analysis.

Before vision models, LLMs were like a person who could only read descriptions of photos. Vision models give them eyes — they can look at the actual image and describe, analyze, and reason about what they see.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Vision models enable LLMs to understand images — analyzing photos, diagrams, screenshots, and documents visually, powering applications from receipt scanning to UI testing. Approach: GPT-4o/Gemini/Claude accept images directly in messages (native multimodal).
2.Alternative: separate vision encoder (CLIP) → projection layer → LLM (LLaVA architecture). Use cases: image captioning, OCR/document analysis, chart interpretation, UI screenshot analysis, visual QA.
3.Image input: URL, base64, or file upload. Resolution matters: higher res = better detail but more tokens.
4.GPT-4o processes images at multiple scales. Tips: ask specific questions, reference regions, combine with text context.

Real Example

Scenario

Receipt scanning: photograph receipt → send to GPT-4o with 'Extract vendor, date, items, and total as JSON' → model reads the image, returns structured data. No separate OCR pipeline needed.

What you would do

In LLM Engineering & APIs, apply Vision Models to this scenario: Receipt scanning: photograph receipt → send to GPT-4o with 'Extract vendor, date, items, and total as JSON' → model reads the image, returns structured data. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Vision Models (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Vision Models happens in the code.

Vision Models

1from openai import OpenAI  # import dependencies2import base64  # import dependencies3
4client = OpenAI()  # create API client5
6def encode_image(path):  # define a reusable function7    with open(path, "rb") as f:8        return base64.b64encode(f.read()).decode()  # return the result9
10response = client.chat.completions.create(  # call the API11    model="gpt-4o",12    messages=[{13        "role": "user",14        "content": [15            {"type": "text", "text": "Describe this image and extract any text you see."},16            {"type": "image_url", "image_url": {17                "url": f"data:image/png;base64,{encode_image('screenshot.png')}"18            }},19        ],20    }],21    max_tokens=500,22)23print(response.choices[0].message.content)  # show output for debugging

Cheat Sheet

Quick recap

quick ref

•type: image_url in content
•base64 encode local images
•model: gpt-4o for vision
•High res = more tokens = more cost
•Ask specific questions about images
•LLaVA for open-source vision

Common Mistakes

✕Sending unnecessarily high-resolution images — wasting tokens
✕Vague questions about images — 'what is this?' vs specific asks
✕Assuming perfect OCR — always validate extracted data
✕Not handling image format/size API limits