Run Your First Local AI Model in 15 Minutes (Ollama Beginner Guide)

What You’ll Need

• A computer with at least 8GB RAM (16GB recommended)
• 5GB free disk space
• Windows 10/11, macOS, or Linux
• 15 minutes

No GPU required. A GPU makes things faster, but CPU inference works fine for smaller models.

Step 1: Install Ollama (2 minutes)

Ollama is the easiest way to run local LLMs. It handles model downloading, quantization, and serving — all from a single command.

Windows

1. Go to ollama.com
2. Download the Windows installer
3. Run the .exe — it installs and starts automatically
4. Ollama runs as a background service

macOS

1. Go to ollama.com
2. Download the Mac installer
3. Drag to Applications
4. Launch Ollama — it runs in the menu bar

Linux

1

curl -fsSL https://ollama.com/install.sh | sh

Verify installation:

1

ollama --version

Step 2: Pull Your First Model (3 minutes)

Ollama models are pre-quantized and optimized for consumer hardware. Let’s start with a small, capable model:

1

ollama pull llama3.2:3b

This downloads a 3-billion-parameter Llama 3.2 model. It’s about 2GB — small enough to run on almost anything with 8GB RAM.

What just happened? Ollama downloaded the model in GGUF format (quantized to 4-bit), stored it locally, and made it available for inference. The model file lives at ~/.ollama/models/ (or %USERPROFILE%\.ollama\models\ on Windows).

Step 3: Chat with Your Model (1 minute)

1

ollama run llama3.2:3b

You’re now chatting with a local AI model. Try:

1
2
3

>>> Write a Python function to check if a number is prime
>>> What's the difference between TCP and UDP?
>>> Explain quantization like I'm five

Type /bye to exit the chat.

No internet required. The model runs entirely on your machine. Disconnect your network and it still works.

Step 4: Try a Bigger Model (5 minutes)

If your machine has 16GB+ RAM or a GPU with 6GB+ VRAM, try a larger model:

1
2
3
4
5
6
7
8

# 8B model — better reasoning, needs 8GB RAM
ollama pull llama3.1:8b

# 7B model — excellent general purpose
ollama pull qwen2.5:7b

# 14B model — needs 16GB RAM or 12GB VRAM
ollama pull qwen2.5:14b

How to Choose a Model

Step 5: Use the API (2 minutes)

Ollama runs an OpenAI-compatible API server on localhost:11434. You can use it with any tool that supports the OpenAI API:

1
2
3
4
5
6

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [
    {"role": "user", "content": "What is self-hosted AI?"}
  ]
}'

Or in Python:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import requests

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "llama3.2:3b",
        "messages": [{"role": "user", "content": "Explain GGUF quantization"}],
        "stream": False
    }
)

print(response.json()["message"]["content"])

Step 6: Add a Web UI (2 minutes)

The command line is fine, but a web interface is nicer. Install Open WebUI (formerly Ollama WebUI):

With Docker:

1
2
3
4
5

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 in your browser. You now have a ChatGPT-like interface running entirely locally.

Without Docker:

1
2

pip install open-webui
open-webui serve

Troubleshooting

“Out of memory” errors

• Use a smaller model (3B instead of 8B)
• Close other memory-heavy applications
• If you have a GPU, ensure Ollama is using it: ollama ps

Slow responses on CPU

• CPU inference is 10-50x slower than GPU
• Stick to 3B-7B models on CPU
• For faster CPU inference, try llama.cpp with optimized builds

Model not using GPU

1
2
3
4
5

# Check what's running
ollama ps

# Force GPU
OLLAMA_GPU_OVERHEAD=0 ollama run llama3.2:3b

What’s Next?

• Best GPUs for Local LLMs in 2026 — Upgrade your hardware
• Ollama vs LM Studio: Which Should You Choose? — Compare tools
• Setting Up ComfyUI for Local Image Generation — Add image generation to your stack

Use the VRAM Calculator to find the right model size for your GPU.