Multimodal

Real-world tasks involve multiple data types: analyze a chart in a PDF, describe a video, answer questions about a photo. Multimodal models handle all inputs in one model, eliminating brittle multi-model pipelines.

A doctor who can listen to symptoms (audio), read lab results (text), examine X-rays (images), and watch patient movement (video) — all integrated into one diagnosis, not four separate specialists.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Multimodal AI processes and generates across text, images, audio, and video in unified models — enabling rich applications that mirror how humans perceive the world through multiple senses.
2.Native multimodal: GPT-4o, Gemini 2.0, Claude (vision) — trained on interleaved modalities from start.
3.Architecture: unified token space where image patches, audio frames, and text tokens flow through same transformer.
4.Use cases: document analysis (text+images), video understanding, visual QA, multimodal RAG (index images+text), accessibility.
5.API pattern: content array with mixed type entries (text, image_url, input_audio).

Real Example

Scenario

Product support agent: customer sends screenshot of error + describes problem in text. GPT-4o analyzes screenshot (reads error message, identifies UI state) combined with text description → provides targeted fix. Single model, no separate OCR + LLM pipeline.

What you would do

In LLM Engineering & APIs, apply Multimodal to this scenario: Product support agent: customer sends screenshot of error + describes problem in text. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Multimodal (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Multimodal happens in the code.

Multimodal

1from openai import OpenAI  # import dependencies2import base64  # import dependencies3
4client = OpenAI()  # create API client5
6# Multimodal: text + image + structured request7response = client.chat.completions.create(  # call the API8    model="gpt-4o",9    messages=[{10        "role": "user",11        "content": [12            {"type": "text", "text": "Analyze this dashboard screenshot. What trends do you see? Any anomalies?"},13            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64.b64encode(open('dashboard.png','rb').read()).decode()}"}},14        ],15    }],16)17
18# Gemini multimodal: text + video19# import google.generativeai as genai20# model = genai.GenerativeModel("gemini-2.0-flash")21# response = model.generate_content(["Summarize this video:", video_file])

Cheat Sheet

Quick recap

quick ref

•content: [{type:text}, {type:image_url}]
•GPT-4o: text + image + audio
•Gemini: + video natively
•Images ≈ 765 tokens each
•Multimodal RAG: CLIP + text embeddings
•Resize images to reduce cost

Common Mistakes

✕Using separate OCR + LLM when native multimodal suffices
✕Sending full-resolution images — expensive and unnecessary
✕Not accounting for multimodal token costs in pricing
✕Assuming all models handle all modalities equally