Ollama

Cloud APIs send data to third parties, cost money per token, and require internet. Ollama lets developers run Llama, Mistral, Gemma, and other models locally with a simple CLI and API — zero cost, full privacy, offline capable.

Ollama is like Spotify offline mode vs streaming — you download the model once, then use it anytime without internet or per-use fees. Trade-off: you need the 'device' (GPU/RAM) to run it.

Visual Workflows

Start here — scroll inside each diagram frame to explore, then use + / − to zoom up to 200% if needed.

Overview

100%

Loading diagram...

Scroll inside the frame to explore · use + / − to zoom up to 200%

100%

Loading diagram...

Scroll inside the frame to explore · use + / − to zoom up to 200%

Local Dev Loop

100%

Loading diagram...

Scroll inside the frame to explore · use + / − to zoom up to 200%

Pull once, then point the same OpenAI client at localhost.

Key Takeaways

1.Ollama is a local LLM runtime that makes it easy to download, run, and interact with open-source models on your own hardware — essential for privacy, offline use, and development without API costs. Ollama wraps llama.cpp and other inference engines.
2.Models stored locally in ~/.ollama/models. CLI: ollama pull, ollama run, ollama list.
3.REST API on localhost:11434 — compatible with OpenAI SDK via base_url override. Modelfiles customize system prompts, parameters, and templates.
4.Supports: Llama 3, Mistral, Gemma, Phi, Qwen, and more in quantized formats (Q4_K_M default). Hardware: needs sufficient RAM/VRAM — 8B model ~5GB, 70B model ~40GB quantized.

Real Example

Scenario

Development workflow: ollama pull llama3.2 → ollama run llama3.2 for interactive chat → point your app's OpenAI client to http://localhost:11434/v1 → develop and test agents locally for free before deploying with cloud APIs.

What you would do

In LLM Engineering & APIs, apply Ollama to this scenario: Development workflow: ollama pull llama3. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Ollama (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Ollama happens in the code.

Ollama

1# CLI usage2# ollama pull llama3.23# ollama run llama3.2 "Explain transformers"4
5# Python with OpenAI SDK pointed at Ollama6from openai import OpenAI  # import dependencies7
8client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # create API client9
10response = client.chat.completions.create(  # call the API11    model="llama3.2",12    messages=[{"role": "user", "content": "What is RAG?"}],13)14print(response.choices[0].message.content)  # show output for debugging15
16# Direct Ollama API17import requests  # import dependencies18resp = requests.post("http://localhost:11434/api/chat", json={  # key line for Ollama19    "model": "llama3.2",20    "messages": [{"role": "user", "content": "Hello!"}],21    "stream": False,22})23print(resp.json()["message"]["content"])  # show output for debugging

Commands

Commands to Remember

curl -fsSL https://ollama.com/install.sh | sh # install Ollama
ollama pull llama3.2 # download a quantized model
ollama run llama3.2 # chat in the terminal
OpenAI(base_url='http://localhost:11434/v1', api_key='ollama') # same SDK, local model

Cheat Sheet

Quick recap

quick ref

•ollama pull llama3.2
•ollama run llama3.2
•API: localhost:11434
•OpenAI SDK: base_url=localhost:11434/v1
•8B Q4 ≈ 5GB RAM
•Modelfile for customization

Common Mistakes

✕Trying to run 70B models on 16GB RAM — out of memory
✕Using local models for production without latency testing
✕Not quantizing — full FP16 models too large for consumer hardware
✕Forgetting ollama serve must be running for API access